On 30 May 2012 14:25, Dan Berindei <dan.berin...@gmail.com> wrote: > On Wed, May 30, 2012 at 4:12 PM, Manik Surtani <ma...@jboss.org> wrote: >> >> On 30 May 2012, at 13:55, Sanne Grinovero wrote: >> >>> On 30 May 2012 13:36, Manik Surtani <ma...@jboss.org> wrote: >>>> Well, one problem leads to another, as you are well aware. Flaky parallel >>>> suite leads careless commits. Yes, we should fix what is broken at the >>>> moment but that is not enough since it will get this way again unless we >>>> have a stable suite that can be used to ensure quality moving fwd. The >>>> options are: >>>> >>>> 1. We always use the sequential suite. >>>> 2. We identify tests that are reliable with the parallel suite (should be >>>> at least 90% of the tests IMO) and use this, leaving the sequential suite >>>> to BuildHive. >>> >>> Ah, now I understand. You want to have some tests executed only when >>> run sequentially.. that looks like a nice compromise, cheers! >>> that will also speed up our test runs, as not all tests are executed; >>> we could exclude also some of the slower ones even if they could work >>> fine in parallel. >> >> Precisely. No loss in quality, just changing the order of delivery of >> information to the reviewer. :) >> > > Ok, first on the list of sequential-only tests: everything that uses > Arquillian :) > I was looking at the cdi test failures and it looks like Arquillian > uses a thread-local variable and that is initialized only on the > thread that calls their @BeforeSuite method. So every time > Arquillian.arquillianBefore/AfterClass runs in a different thread, it > throws a NullPointerException. > > I see two issues with your plan, though: > 1. Buildhive is limited to 15 mins, and a reviewer wouldn't > necessarily wait for 2 hours to integrate a pull request anyway. So > the sequential build would be limited to Jenkins runs. > 2. How do we select which tests run where? I remember we had to > disable tests precisely because configuring test groups in > testng/surefire didn't work.
I wouldn't oppose a good plan just because of some minor technical difficulties. We can solve those in many ways. # improve Arquillian # make-your-own BuildHive by using same integration / have them lift limitations. (i.e. trust Galder and give him some time..) # fix/workaround TestNG / switch to JUnit / make your own Sanne _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
