Looks like this test case is a recent addition: https://issues.apache.org/jira/browse/LUCENE-10292
And I agree -- FreeTextSuggester and the Lookup javadocs need some shoring up if we want to indeed support suggestions during rebuild. Mike McCandless http://blog.mikemccandless.com On Wed, Apr 27, 2022 at 7:51 AM Dawid Weiss <[email protected]> wrote: > I looked at the test that failed here - > TestFreeTextSuggester.testLookupsDuringReBuild and for the life of it > I can't see how it's even supposed to work. What it seems to be > testing is a concurrent rebuild of the suggester while suggestions are > still being served. But the code inside FreeTextSuggester has no means > whatsoever to make this happen - count and fst fields are just updated > without any synchronization or even caution about concurrency. In > fact, I don't see anything in the Lookup interface that would suggest > it's safe to rebuild a suggester concurrently: > > /** > * Builds up a new internal {@link Lookup} representation based on the > given {@link > * InputIterator}. The implementation might re-sort the data internally. > */ > public abstract void build(InputIterator inputIterator) throws IOException; > > Is this something that was added in between? Is this even enforced > anywhere? Seems like a buggy test that verifies something that isn't > guaranteed. > > Dawid > > On Wed, Apr 27, 2022 at 10:45 AM Policeman Jenkins Server > <[email protected]> wrote: > > > > Build: https://jenkins.thetaphi.de/job/Lucene-9.x-MacOSX/499/ > > Java: 64bit/jdk-16.0.2 -XX:+UseCompressedOops -XX:+UseParallelGC > > > > 3 tests failed. > > FAILED: > org.apache.lucene.search.suggest.analyzing.TestFreeTextSuggester.testLookupsDuringReBuild > > > > Error Message: > > java.lang.AssertionError: expected:<1> but was:<0> > > > > Stack Trace: > > java.lang.AssertionError: expected:<1> but was:<0> > > at > __randomizedtesting.SeedInfo.seed([CA3DBCCF7F252296:106E670A4E65E420]:0) > > at [email protected]/org.junit.Assert.fail(Assert.java:89) > > at [email protected]/org.junit.Assert.failNotEquals(Assert.java:835) > > at [email protected]/org.junit.Assert.assertEquals(Assert.java:647) > > at [email protected]/org.junit.Assert.assertEquals(Assert.java:633) > > at > org.apache.lucene.search.suggest.analyzing.TestFreeTextSuggester.lambda$testLookupsDuringReBuild$0(TestFreeTextSuggester.java:105) > > at > org.apache.lucene.search.suggest.SuggestRebuildTestUtil.testLookupsDuringReBuild(SuggestRebuildTestUtil.java:75) > > at > org.apache.lucene.search.suggest.analyzing.TestFreeTextSuggester.testLookupsDuringReBuild(TestFreeTextSuggester.java:101) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:78) > > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:567) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44) > > at [email protected] > /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at [email protected] > /org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at [email protected] > /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at [email protected] > /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) > > at [email protected] > /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819) > > at [email protected] > /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887) > > at [email protected] > /com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898) > > at [email protected] > /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at [email protected] > /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) > > at [email protected] > /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at [email protected] > /com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at [email protected] > /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at [email protected] > /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > > at [email protected] > /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at [email protected] > /org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) > > at [email protected] > /org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at [email protected] > /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at [email protected] > /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370) > > at [email protected] > /com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826) > > at java.base/java.lang.Thread.run(Thread.java:831) > > > > > > FAILED: > org.apache.lucene.search.suggest.analyzing.TestFreeTextSuggester.classMethod > > > > Error Message: > > com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from > SUITE scope at > org.apache.lucene.search.suggest.analyzing.TestFreeTextSuggester: > > 1) Thread[id=23, name=Thread-3, state=WAITING, > group=TGRP-TestFreeTextSuggester] > > at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) > > at [email protected] > /java.util.concurrent.locks.LockSupport.park(LockSupport.java:211) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:714) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1026) > > at [email protected] > /java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:341) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$DelayedIterator.next(SuggestRebuildTestUtil.java:116) > > at > app//org.apache.lucene.search.suggest.InputArrayIterator.next(InputArrayIterator.java:66) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:279) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:237) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil.lambda$testLookupsDuringReBuild$0(SuggestRebuildTestUtil.java:66) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$$Lambda$256/0x0000000800e12608.run(Unknown > Source) > > at [email protected]/java.lang.Thread.run(Thread.java:831) > > > > Stack Trace: > > com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from > SUITE scope at > org.apache.lucene.search.suggest.analyzing.TestFreeTextSuggester: > > 1) Thread[id=23, name=Thread-3, state=WAITING, > group=TGRP-TestFreeTextSuggester] > > at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) > > at [email protected] > /java.util.concurrent.locks.LockSupport.park(LockSupport.java:211) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:714) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1026) > > at [email protected] > /java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:341) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$DelayedIterator.next(SuggestRebuildTestUtil.java:116) > > at > app//org.apache.lucene.search.suggest.InputArrayIterator.next(InputArrayIterator.java:66) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:279) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:237) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil.lambda$testLookupsDuringReBuild$0(SuggestRebuildTestUtil.java:66) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$$Lambda$256/0x0000000800e12608.run(Unknown > Source) > > at [email protected]/java.lang.Thread.run(Thread.java:831) > > at __randomizedtesting.SeedInfo.seed([CA3DBCCF7F252296]:0) > > > > > > FAILED: > org.apache.lucene.search.suggest.analyzing.TestFreeTextSuggester.classMethod > > > > Error Message: > > com.carrotsearch.randomizedtesting.ThreadLeakError: There are still > zombie threads that couldn't be terminated: > > 1) Thread[id=23, name=Thread-3, state=WAITING, > group=TGRP-TestFreeTextSuggester] > > at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) > > at [email protected] > /java.util.concurrent.locks.LockSupport.park(LockSupport.java:211) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:714) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1026) > > at [email protected] > /java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:341) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$DelayedIterator.next(SuggestRebuildTestUtil.java:116) > > at > app//org.apache.lucene.search.suggest.InputArrayIterator.next(InputArrayIterator.java:66) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:279) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:237) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil.lambda$testLookupsDuringReBuild$0(SuggestRebuildTestUtil.java:66) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$$Lambda$256/0x0000000800e12608.run(Unknown > Source) > > at [email protected]/java.lang.Thread.run(Thread.java:831) > > > > Stack Trace: > > com.carrotsearch.randomizedtesting.ThreadLeakError: There are still > zombie threads that couldn't be terminated: > > 1) Thread[id=23, name=Thread-3, state=WAITING, > group=TGRP-TestFreeTextSuggester] > > at [email protected]/jdk.internal.misc.Unsafe.park(Native Method) > > at [email protected] > /java.util.concurrent.locks.LockSupport.park(LockSupport.java:211) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:714) > > at [email protected] > /java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1026) > > at [email protected] > /java.util.concurrent.Semaphore.acquireUninterruptibly(Semaphore.java:341) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$DelayedIterator.next(SuggestRebuildTestUtil.java:116) > > at > app//org.apache.lucene.search.suggest.InputArrayIterator.next(InputArrayIterator.java:66) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:279) > > at > app//org.apache.lucene.search.suggest.analyzing.FreeTextSuggester.build(FreeTextSuggester.java:237) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil.lambda$testLookupsDuringReBuild$0(SuggestRebuildTestUtil.java:66) > > at > app//org.apache.lucene.search.suggest.SuggestRebuildTestUtil$$Lambda$256/0x0000000800e12608.run(Unknown > Source) > > at [email protected]/java.lang.Thread.run(Thread.java:831) > > at __randomizedtesting.SeedInfo.seed([CA3DBCCF7F252296]:0) > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
