On 30 May 2012 13:36, Manik Surtani <ma...@jboss.org> wrote: > Well, one problem leads to another, as you are well aware. Flaky parallel > suite leads careless commits. Yes, we should fix what is broken at the > moment but that is not enough since it will get this way again unless we have > a stable suite that can be used to ensure quality moving fwd. The options > are: > > 1. We always use the sequential suite. > 2. We identify tests that are reliable with the parallel suite (should be at > least 90% of the tests IMO) and use this, leaving the sequential suite to > BuildHive.
Ah, now I understand. You want to have some tests executed only when run sequentially.. that looks like a nice compromise, cheers! that will also speed up our test runs, as not all tests are executed; we could exclude also some of the slower ones even if they could work fine in parallel. Sanne > > My pref is #2 but that requires identifying the tests that flake out with the > parallel suite. > > > > On 30 May 2012, at 13:25, Sanne Grinovero wrote: > >> It's not really about "which" test mode we should use, as long as >> people check their commits and have any test that gives enough >> confidence, which is still not the case even during this discussion: >> >> last build Infinispan-master-JDK6-tcp-NON_PARALLEL : 123 failures / +95 >> failures >> >> We need to use non-parallel if that's the only choice we currently >> have, until parallel is fixed too. I agree that's time consuming, but >> that's exactly why I'm worried about the fact we leave this go this >> far. Also it doesn't really take 2 hours anymore, and slow tests can >> be speed up in many ways.. yes that's an effort, but it will pay off. >> >> Cheers, >> Sanne >> >> >> >> On 30 May 2012 13:00, Dan Berindei <dan.berin...@gmail.com> wrote: >>> On Tue, May 29, 2012 at 1:48 PM, Manik Surtani <ma...@jboss.org> wrote: >>>> I pretty much agree with this; and here's a bit of history. >>>> >>>> For the large part we have had a stable test suite, but the occasional >>>> unpredictability in the suite came in when we introduced the parallel test >>>> runner, to allow us to run the (core) suite in under 5 minutes - a suite >>>> which otherwise took over 2 hours when run sequentially. >>>> >>>> We could revert back to just using the sequential test runner if people >>>> prefer that - it makes the suite run more predictably and hence easier to >>>> debug and maintain - but the drawback is, well, it takes 2 hours to run. >>>> >>>> Perhaps we should use the parallel suite as a "smoke test", but in the >>>> event of any failures, revert to a run using the sequential suite? >>>> >>> >>> -1, a smoke test should be something that is not only faster but >>> always passes, so we could run that on each pull req. Getting a FAIL >>> from buildhive on each pull request would get tiring real quick. >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Manik Surtani > ma...@jboss.org > twitter.com/maniksurtani > > Lead, Infinispan > http://www.infinispan.org > > > > > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
