Thanks a lot for your information & insights, I will try to reproduce the errors and investigate the results. And, maybe I should learn more about internal of the test framework, I'm not familiar with it and still do not understand what does "seed" means exactly in this context.
Regards, Tomoko 2018年8月22日(水) 1:05 Erick Erickson <erickerick...@gmail.com>: > Couple of things (and I know you've been around for a while, so pardon > me if it's all old hat to you): > > 1> if you run the entire "reproduce with" line and can get a > consistent failure, then you are half way there, nothing is as > frustrating as not getting failures reliably. The critical bit is > often the -Dtests.seed. As Michael mentioned, there are various > randomizations done for _many_ things in Lucene tests using a random > generator. tests.seed, well, seeds that generator so it produces the > same numbers every time it's run with that seed. You'll see lots of > calls to a static ramdom() method calls. I'll add that if you want to > use randomness in your tests, use that method and do _not_ use a local > instance of Java's Random. > > 2> MIke: You say IntelliJ succeeds. But that'll use a new random() > seed. Once you run a test, in the upper right (on my version at > least), IntelliJ will show you a little box with the test name and you > can "edit configurations" on it. I often have luck by editing the > configuration and adding the test seed to the "VM option" box for the > test, just the "-Dtests.seed=35AF58F652536895" part. You can add all > of the -D flags in the "reproduce with" line if you want, but often > just the seed works for me. If that works and you track it down, do > remember to take that seed _out_ of the "VM options" box rather than > forget it as I have done ;) > > 3> Mark Miller's beasting script can be used to run a zillion tests > over night: https://gist.github.com/markrmiller/dbdb792216dc98b018ad > > 4> If MIke's intuition that it's one of the file system randomizations > that occasionally gets hit _and_ you determine that that's an invalid > test case (and for Luke requiring that the FS-basesd tests are all > that are necessary may be fine) I'm pretty sure you you can disable > that randomization for your specific tests. > > Best, > Erick > > On Tue, Aug 21, 2018 at 7:47 AM, Tomoko Uchida > <tomoko.uchida.1...@gmail.com> wrote: > > Hi, Mike > > > > Thanks for sharing your experiments. > > > >> CommitsImplTest.testListCommits > >> CommitsImplTest.testGetCommit_generation_notfound > >> CommitsImplTest.testGetSegments > >> DocumentsImplTest.testGetDocumentFIelds > > > > I also found CommitsImplTest and DocumentsImplTest fail frequently, > > especially CommitsImplTest is unhappy with lucene test framework (I > pointed > > that in my previous post.) > > > >> I wonder if this is somehow related to running mvn from command line vs > > running in IntelliJ since previously I was doing the latter > > > > In my personal experience, when I was running those suspicious tests on > > IntelliJ IDEA, they were always green - but I am not sure that `mvn test` > > is the cause. > > > > Thanks, > > Tomoko > > > > 2018年8月21日(火) 22:53 Michael Sokolov <msoko...@gmail.com>: > > > >> I was running these luke tests a bunch and found the following tests > fail > >> intermittently; pretty frequently. Once I @Ignore them I can get a > >> consistent pass: > >> > >> > >> CommitsImplTest.testListCommits > >> CommitsImplTest.testGetCommit_generation_notfound > >> CommitsImplTest.testGetSegments > >> DocumentsImplTest.testGetDocumentFIelds > >> > >> I did not attempt to figure out why the tests were failing, but to do > that, > >> I would: > >> > >> Run repeatedly until you get a failure -- save the test "seed" from this > >> run that should be printed out in the failure message Then you should be > >> able to reliably reproduce this failure by re-running with system > property > >> "tests.seed" set to that value. This is used to initialize the > >> randomization that LuceneTestCase does. > >> > >> My best guess is that the failures may have to do with randomly using > some > >> Directory implementation or other Lucene feature that Luke doesn't > properly > >> handle? > >> > >> Hmm I was trying this again to see if I could get an example, and > strangely > >> these tests are no longer failing for me after several runs, when > >> previously they failed quite often. I wonder if this is somehow related > to > >> running mvn from command line vs running in IntelliJ since previously I > was > >> doing the latter > >> > >> -Mike > >> > >> On Tue, Aug 21, 2018 at 9:01 AM Tomoko Uchida < > >> tomoko.uchida.1...@gmail.com> > >> wrote: > >> > >> > Hello, > >> > > >> > Could you give me some advice or comments about usage of > LuceneTestCase. > >> > > >> > Some of our unit tests extending LuceneTestCase fail by assertion > error > >> -- > >> > sometimes, randomly. > >> > I suppose we use LuceneTestCase in inappropriate way, but cannot find > out > >> > how to fix it. > >> > > >> > Here is some information about failed tests. > >> > > >> > * The full test code is here: > >> > > >> > > >> > https://github.com/DmitryKey/luke/blob/master/src/test/java/org/apache/lucene/luke/models/commits/CommitsImplTest.java > >> > * We run tests by `mvn test` on Mac PC or Travis CI (oracle > jdk8/9/10, > >> > openjdk 8/9/10), assertion errors occur regardless of platform or jdk > >> > version. > >> > * Stack trace of an assertion error is at the end of this mail. > >> > > >> > Any advice are appreciated. Please tell me if more information is > needed. > >> > > >> > Thanks, > >> > Tomoko > >> > > >> > > >> > ------------------------------------------------------- > >> > T E S T S > >> > ------------------------------------------------------- > >> > Running org.apache.lucene.luke.models.commits.CommitsImplTest > >> > NOTE: reproduce with: ant test -Dtestcase=CommitsImplTest > >> > -Dtests.method=testGetSegmentAttributes -Dtests.seed=35AF58F652536895 > >> > -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=de > >> > -Dtests.timezone=Africa/Kigali -Dtests.asserts=true > >> > -Dtests.file.encoding=UTF-8 > >> > NOTE: leaving temporary files on disk at: > >> > > >> > > >> > /private/var/folders/xr/mrs6w1m15y1f4wkgfhn_x1dm0000gp/T/lucene.luke.models.commits.CommitsImplTest_35AF58F652536895-001 > >> > NOTE: test params are: > >> > > >> > > >> > codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, > >> > chunkSize=6, maxDocsPerChunk=7, blockSize=2), > >> > > >> > > >> > termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, > >> > chunkSize=6, blockSize=2)), sim=RandomSimilarity(queryNorm=true): {}, > >> > locale=de, timezone=Africa/Kigali > >> > NOTE: Mac OS X 10.13.6 x86_64/Oracle Corporation 1.8.0_181 > >> > (64-bit)/cpus=4,threads=1,free=201929064,total=257425408 > >> > NOTE: All tests run in this JVM: [CommitsImplTest] > >> > Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.44 > sec > >> > <<< FAILURE! > >> > > >> > > >> > testGetSegmentAttributes(org.apache.lucene.luke.models.commits.CommitsImplTest) > >> > Time elapsed: 0.047 sec <<< FAILURE! > >> > java.lang.AssertionError > >> > at > >> __randomizedtesting.SeedInfo.seed([35AF58F652536895:AE37E8467BC01918]:0) > >> > at org.junit.Assert.fail(Assert.java:92) > >> > at org.junit.Assert.assertTrue(Assert.java:43) > >> > at org.junit.Assert.assertTrue(Assert.java:54) > >> > at > >> > > >> > > >> > org.apache.lucene.luke.models.commits.CommitsImplTest.testGetSegmentAttributes(CommitsImplTest.java:151) > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > at > >> > > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > at > >> > > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:498) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) > >> > at > >> > > >> > > >> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890) > >> > at > >> > > >> > > >> > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) > >> > at > >> > > >> > > >> > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > >> > at > >> > > >> > > >> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) > >> > at java.lang.Thread.run(Thread.java:748) > >> > > >> > > > > > > -- > > Tomoko Uchida > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > -- Tomoko Uchida
