You can also tell the test framework to temporarily skip the bad apple tests with the flag "-Dtests.badapples=false", which is sometimes useful. Just an FYI.
Jason On Fri, Aug 24, 2018 at 11:14 AM Erick Erickson <[email protected]> wrote: > > One other thing: > There are two sources of randomness. the difficult one is the > interplay between machine speed, disk speed, CPU speed, all the > gremlins that exist with real hardware. > > The second is deliberate randomness. You'll see calls to random() (a > static method in the test framework) that deliberately does something > different. For instance it might choose a random locale, or index > varying numbers of docs, something like this to index between 5 and 15 > docs.: > > int num = random().nextInt(10) + 5 > for (int idx = 0; idx < num; ++idx) { > index a doc > } > > So when a test fails, in the log you'll see something like "reproduce > with 'ant -Dtestcase=blah -Dtests.seed=2839923749 .....'. Using that > exact line, all the deliberate randomness will be reproduced exactly. > If one of your failing cases fails reliably with a line like above, > please let us know! > > FWIW, > On Fri, Aug 24, 2018 at 8:06 AM Charles Sanders <[email protected]> wrote: > > > > Thanks Erick! Excellent info. > > > > > > ________________________________ > > From: "Erick Erickson" <[email protected]> > > To: [email protected] > > Sent: Friday, August 24, 2018 11:02:15 AM > > Subject: Re: Tests Fail > > > > Unfortunately yes, we've built up considerable "test debt" that we're > > working on, failures occur intermittently based on random factors and > > are pretty hard to track down. See: > > http://fucit.org/solr-jenkins-reports/failure-report.html > > > > There are a few things I do at this point when a test fails: > > 1> rerun the test a couple of times, and if it passes don't worry. Do > > this by, for instance, 'ant -Dtestcase=MoveReplicaHDFSTest test' from > > the root of your install > > 2> look at the above link to see if your test is "featured" > > 3> check the test itself, if it has the @BadApple notation then it's > > been identified as flakey. > > > > If you can ever figure out _why_ some of these tests randomly fail > > that would be fantastic! > > > > Best, > > Erick > > On Fri, Aug 24, 2018 at 6:44 AM Charles Sanders <[email protected]> wrote: > > > > > > Hello, > > > I'm new at this, so its probably something I've done... or not done. I > > > fetched from master this morning and ran 'ant clean test'. But I'm > > > getting test failures. I'm following the directions found here: > > > https://wiki.apache.org/solr/HowToContribute > > > > > > Is this normal to have test failures from master? I see emails from > > > Jenkins that the build is unstable. > > > [JENKINS] Lucene-Solr-repro - Build # 1295 - Still Unstable > > > > > > Just trying to understand what I should expect. Thanks for helping the > > > new guy. > > > -Charles > > > > > > > > > [junit4] Tests with failures [seed: C83CD1AEBC29ADAF]: > > > [junit4] - org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove > > > [junit4] - > > > org.apache.solr.cloud.TestAuthenticationFramework.testBasics > > > [junit4] > > > [junit4] > > > [junit4] JVM J0: 0.89 .. 3088.54 = 3087.65s > > > [junit4] JVM J1: 0.63 .. 3088.48 = 3087.85s > > > [junit4] JVM J2: 0.89 .. 3088.64 = 3087.75s > > > [junit4] Execution time total: 51 minutes 28 seconds > > > [junit4] Tests summary: 829 suites (5 ignored), 4028 tests, 2 errors, > > > 170 ignored (150 assumptions) > > > > > > BUILD FAILED > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
