The problem here is that TopScoreDocCollectorManager creates a priority queue of 250_000 for every collector it creates. This test only ever indexes max 1000 documents. In the case I examined we had 8 segments, which get partitioned into 5 partitions each. That means we will build 40 priority queues of 250_000 items. This is a bit of an extreme case that's far from reality, yet I wonder, is the problem the number of partitions or the size of the priority queue, or both?
On Wed, Sep 11, 2024 at 7:39 AM Luca Cavanna <java...@apache.org> wrote: > This may have to do with intra-segment concurrency. I will take a look. > > On Wed, Sep 11, 2024, 05:39 Policeman Jenkins Server <jenk...@thetaphi.de> > wrote: > >> Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/50111/ >> Java: 64bit/hotspot/jdk-21.0.1 -XX:-UseCompressedOops -XX:+UseSerialGC >> >> 1 tests failed. >> FAILED: >> org.apache.lucene.sandbox.search.TestLargeNumHitsTopDocsCollector.testNoPQBuild >> >> Error Message: >> java.lang.OutOfMemoryError: Java heap space >> >> Stack Trace: >> java.lang.OutOfMemoryError: Java heap space >> at >> __randomizedtesting.SeedInfo.seed([2E3BF44EC3457CCA:DA6EA4532975FABD]:0) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.HitQueue.lambda$new$0(HitQueue.java:70) >> at >> org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.search.HitQueue$$Lambda/0x000071d2ec233150.get(Unknown >> Source) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:105) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.HitQueue.<init>(HitQueue.java:63) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:203) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector.<init>(TopScoreDocCollector.java:50) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.TopScoreDocCollectorManager.newCollector(TopScoreDocCollectorManager.java:140) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.TopScoreDocCollectorManager.newCollector(TopScoreDocCollectorManager.java:28) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:741) >> at org.apache.lucene.core@10.0.0-SNAPSHOT >> /org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:725) >> at >> org.apache.lucene.sandbox.search.TestLargeNumHitsTopDocsCollector.testNoPQBuild(TestLargeNumHitsTopDocsCollector.java:111) >> at >> java.base/java.lang.invoke.LambdaForm$DMH/0x000071d2ec114000.invokeVirtual(LambdaForm$DMH) >> at >> java.base/java.lang.invoke.LambdaForm$MH/0x000071d2ec0bc800.invoke(LambdaForm$MH) >> at >> java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) >> at >> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153) >> at >> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) >> at java.base/java.lang.reflect.Method.invoke(Method.java:580) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >> at junit@4.13.1 >> /org.junit.rules.RunRules.evaluate(RunRules.java:20) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) >> at randomizedtesting.runner@2.8.1 >> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org >> For additional commands, e-mail: builds-h...@lucene.apache.org > >
