This is a test bug, but I'm not yet sure how to fix it. The test verifies the sampling works OK by computing some statistics about the counted facets. In particular it computes the standard deviation and ensures that it's smaller than some arbitrary value (200). However, with this seed and test parameters, the standard deviation is 215, and I've verified that with any seed, if you fix the number of indexed documents to a high enough number (>50,000), it will likely be bigger than 200.
What I'm not sure about is how to fix the test -- increasing the number from 200 to 300 will only push the limit further until another failure, because of other test parameters. I can do that, and investigate again if another run fails. But increasing that number too high misses the point I think, since if our random sampling isn't really random, we'll fail to detect that. Basically, with some very bad luck, we could sample such that we hit the maximum value of the variance, and therefore no matter the value we'll compare the standard deviation to, we might run into this extremely bad-luck-case at some point. One choice is to increase the value now, and accept that some runs may fail, once in a long while ... Do we have other tests that do random sampling of stuff and assert the sampled values? Shai On Mon, Sep 22, 2014 at 10:19 PM, Apache Jenkins Server < [email protected]> wrote: > Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/635/ > > 1 tests failed. > REGRESSION: > org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling > > Error Message: > > > Stack Trace: > java.lang.AssertionError > at > __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0) > at org.junit.Assert.fail(Assert.java:92) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertTrue(Assert.java:54) > at > org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) > at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) > at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) > at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) > at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) > at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) > at java.lang.Thread.run(Thread.java:745) > > > > > Build Log: > [...truncated 8590 lines...] > [junit4] Suite: > org.apache.lucene.facet.TestRandomSamplingFacetsCollector > [junit4] 2> NOTE: download the large Jenkins line-docs file by > running 'ant get-jenkins-line-docs' in the lucene directory. > [junit4] 2> NOTE: reproduce with: ant test > -Dtestcase=TestRandomSamplingFacetsCollector > -Dtests.method=testRandomSampling -Dtests.seed=EB7A704156A4175F > -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true > -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt > -Dtests.locale=nl_NL -Dtests.timezone=VST -Dtests.file.encoding=ISO-8859-1 > [junit4] FAILURE 21.1s J2 | > TestRandomSamplingFacetsCollector.testRandomSampling <<< > [junit4] > Throwable #1: java.lang.AssertionError > [junit4] > at > __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0) > [junit4] > at > org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136) > [junit4] > at java.lang.Thread.run(Thread.java:745) > [junit4] 2> NOTE: test params are: codec=Lucene410: > {$payloads$=FST41, $full_path$=FST41, $facets=FSTOrd41, > EvenOdd=PostingsFormat(name=Memory doPackFST= false)}, > docValues:{$facets=DocValuesFormat(name=Lucene410)}, > sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {EvenOdd=DFR > I(F)L1}, locale=nl_NL, timezone=VST > [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation > 1.7.0_65 (64-bit)/cpus=16,threads=1,free=197229168,total=297795584 > [junit4] 2> NOTE: All tests run in this JVM: > [TestOrdinalMappingAtomicReader, TestCachedOrdinalsReader, > TestRandomSamplingFacetsCollector] > [junit4] Completed on J2 in 21.20s, 1 test, 1 failure <<< FAILURES! > > [...truncated 58 lines...] > BUILD FAILED > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:498: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:471: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:60: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:462: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:2140: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/module-build.xml:58: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358: > The following error occurred while executing this line: > /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: > There were test failures: 23 suites, 155 tests, 1 failure > > Total time: 116 minutes 23 seconds > Build step 'Invoke Ant' marked build as failure > Archiving artifacts > Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #633 > Archived 5 artifacts > Archive block size is 32768 > Received 0 blocks and 228817 bytes > Compression is 0.0% > Took 0.11 sec > Recording test results > Email was triggered for: Failure > Sending email for trigger: Failure > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
