IMO, test groups + maven integration to run this = PITA On May 30, 2012, at 6:22 PM, Dan Berindei wrote:
> On Wed, May 30, 2012 at 5:19 PM, Manik Surtani <ma...@jboss.org> wrote: >> >> On 30 May 2012, at 14:58, Sanne Grinovero wrote: >> >> >> I see two issues with your plan, though: >> >> 1. Buildhive is limited to 15 mins, and a reviewer wouldn't >> >> necessarily wait for 2 hours to integrate a pull request anyway. So >> >> the sequential build would be limited to Jenkins runs. >> >> 2. How do we select which tests run where? I remember we had to >> >> disable tests precisely because configuring test groups in >> >> testng/surefire didn't work. >> >> >> I wouldn't oppose a good plan just because of some minor technical >> difficulties. We can solve those in many ways. >> # improve Arquillian >> >> >> Yup. >> > > That would be nice, but not vital as long as we have a solution to > exclude Arquillian tests from manual/buildhive runs. > >> # make-your-own BuildHive by using same integration / have them lift >> limitations. (i.e. trust Galder and give him some time..) >> >> >> Yup. Galder is working with CloudBees to lift this restriction. We could >> also always ping our friends at OpenShift. >> > > Still, it doesn't make sense to wait 2 hours every time after every > pull request update to see that it's fine in buildhive. So I imagine > buildhive will continue running tests in parallel. > >> # fix/workaround TestNG / switch to JUnit / make your own >> > > At some point it's simpler to just fix the tests... > >> >> Test groups do work, just overriding them on the command line was flaky. >> The correct approach is to use Maven profiles. Solution: fix Maven POMs. >> > > Hmm, I see our POMs already have different profiles setting the test > groups, and maven really doesn't run manual tests by default. Nice! > > Yet there are still lots of tests that have both group="manual" and > enabled="false", with descriptions like "Disabled until we can > configure Surefire to skip manual tests". You can't blame me for > thinking the comment is still valid :) > > So now I have a counter-suggestion: > 1. Add a 'flaky-' prefix to the group name of tests that fail randomly > (or always!) instead of disabling them. > 2. Always run tests in parallel, forget about sequential runs as they > take way too long. Many of the changes we'd have to make so that tests > take less time sequentially will make it easier for those tests to > fail when they run in parallel. > 3. Configure a separate build in jenkins to run flaky tests as well, > let everyone else run only non-flaky ones. > 4. Create a JIRA for fixing flaky tests in general, create separate > JIRAs only where the fix changes production code. > 5. Profit! > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
