I've tried to change the test to assert better that the facet counts are sampled OK. E.g. the test currently computes a STDDEV, but that's wrong since we have only 5 categories that are sampled, and so it's not a real normal distribution, and every difference from the mean results in a very high STDDEV... So I tried Pearson's Chi-Squared test ( http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test), which works better on more runs, but still fails from time to time.
At that point I started thinking what exactly are we testing -- we have a Random sampler which samples DOCUMENTS not CATEGORIES. So e.g. if I index documents w/ categories in that order: A B C D A B C D A B C D and so forth, and I sample every 4th DOCUMENT, I could very well count only category D, while A, B and C might see none to very low counts, resulting in big STDDEV, or Chi-Square result. The sampler has no knowledge of the indexed categories, only the indexed documents. And you could easily write an adversary indexer which indexes categories such that the random sampler almost never samples some categories. Instead I think we should assert that the sampler sampled roughly e.g. 10% of the docs, irrespective of the counts of the categories. What do you think? On Tue, Sep 23, 2014 at 9:47 PM, Shai Erera <[email protected]> wrote: > This is a test bug, but I'm not yet sure how to fix it. The test verifies > the sampling works OK by computing some statistics about the counted > facets. In particular it computes the standard deviation and ensures that > it's smaller than some arbitrary value (200). However, with this seed and > test parameters, the standard deviation is 215, and I've verified that with > any seed, if you fix the number of indexed documents to a high enough > number (>50,000), it will likely be bigger than 200. > > What I'm not sure about is how to fix the test -- increasing the number > from 200 to 300 will only push the limit further until another failure, > because of other test parameters. I can do that, and investigate again if > another run fails. > > But increasing that number too high misses the point I think, since if our > random sampling isn't really random, we'll fail to detect that. > > Basically, with some very bad luck, we could sample such that we hit the > maximum value of the variance, and therefore no matter the value we'll > compare the standard deviation to, we might run into this extremely > bad-luck-case at some point. > > One choice is to increase the value now, and accept that some runs may > fail, once in a long while ... > > Do we have other tests that do random sampling of stuff and assert the > sampled values? > > Shai > > On Mon, Sep 22, 2014 at 10:19 PM, Apache Jenkins Server < > [email protected]> wrote: > >> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/635/ >> >> 1 tests failed. >> REGRESSION: >> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling >> >> Error Message: >> >> >> Stack Trace: >> java.lang.AssertionError >> at >> __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0) >> at org.junit.Assert.fail(Assert.java:92) >> at org.junit.Assert.assertTrue(Assert.java:43) >> at org.junit.Assert.assertTrue(Assert.java:54) >> at >> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) >> at >> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) >> at >> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) >> at >> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) >> at >> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at >> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) >> at java.lang.Thread.run(Thread.java:745) >> >> >> >> >> Build Log: >> [...truncated 8590 lines...] >> [junit4] Suite: >> org.apache.lucene.facet.TestRandomSamplingFacetsCollector >> [junit4] 2> NOTE: download the large Jenkins line-docs file by >> running 'ant get-jenkins-line-docs' in the lucene directory. >> [junit4] 2> NOTE: reproduce with: ant test >> -Dtestcase=TestRandomSamplingFacetsCollector >> -Dtests.method=testRandomSampling -Dtests.seed=EB7A704156A4175F >> -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true >> -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt >> -Dtests.locale=nl_NL -Dtests.timezone=VST -Dtests.file.encoding=ISO-8859-1 >> [junit4] FAILURE 21.1s J2 | >> TestRandomSamplingFacetsCollector.testRandomSampling <<< >> [junit4] > Throwable #1: java.lang.AssertionError >> [junit4] > at >> __randomizedtesting.SeedInfo.seed([EB7A704156A4175F:162195CE7F3E0E8]:0) >> [junit4] > at >> org.apache.lucene.facet.TestRandomSamplingFacetsCollector.testRandomSampling(TestRandomSamplingFacetsCollector.java:136) >> [junit4] > at java.lang.Thread.run(Thread.java:745) >> [junit4] 2> NOTE: test params are: codec=Lucene410: >> {$payloads$=FST41, $full_path$=FST41, $facets=FSTOrd41, >> EvenOdd=PostingsFormat(name=Memory doPackFST= false)}, >> docValues:{$facets=DocValuesFormat(name=Lucene410)}, >> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {EvenOdd=DFR >> I(F)L1}, locale=nl_NL, timezone=VST >> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation >> 1.7.0_65 (64-bit)/cpus=16,threads=1,free=197229168,total=297795584 >> [junit4] 2> NOTE: All tests run in this JVM: >> [TestOrdinalMappingAtomicReader, TestCachedOrdinalsReader, >> TestRandomSamplingFacetsCollector] >> [junit4] Completed on J2 in 21.20s, 1 test, 1 failure <<< FAILURES! >> >> [...truncated 58 lines...] >> BUILD FAILED >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:498: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:471: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:60: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:462: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:2140: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/module-build.xml:58: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1358: >> The following error occurred while executing this line: >> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: >> There were test failures: 23 suites, 155 tests, 1 failure >> >> Total time: 116 minutes 23 seconds >> Build step 'Invoke Ant' marked build as failure >> Archiving artifacts >> Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #633 >> Archived 5 artifacts >> Archive block size is 32768 >> Received 0 blocks and 228817 bytes >> Compression is 0.0% >> Took 0.11 sec >> Recording test results >> Email was triggered for: Failure >> Sending email for trigger: Failure >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > >
