I opened https://github.com/apache/lucene/pull/13775 with a proposed solution.
On Thu, Sep 12, 2024 at 11:07 AM Luca Cavanna <java...@apache.org> wrote: > The problem here is that TopScoreDocCollectorManager creates a priority > queue of 250_000 for every collector it creates. This test only ever > indexes max 1000 documents. In the case I examined we had 8 segments, which > get partitioned into 5 partitions each. That means we will build 40 > priority queues of 250_000 items. This is a bit of an extreme case that's > far from reality, yet I wonder, is the problem the number of partitions or > the size of the priority queue, or both? > > On Wed, Sep 11, 2024 at 7:39 AM Luca Cavanna <java...@apache.org> wrote: > >> This may have to do with intra-segment concurrency. I will take a look. >> >> On Wed, Sep 11, 2024, 05:39 Policeman Jenkins Server <jenk...@thetaphi.de> >> wrote: >> >>> Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/50111/ >>> Java: 64bit/hotspot/jdk-21.0.1 -XX:-UseCompressedOops -XX:+UseSerialGC >>> >>> 1 tests failed. >>> FAILED: >>> org.apache.lucene.sandbox.search.TestLargeNumHitsTopDocsCollector.testNoPQBuild >>> >>> Error Message: >>> java.lang.OutOfMemoryError: Java heap space >>> >>> Stack Trace: >>> java.lang.OutOfMemoryError: Java heap space >>> at >>> __randomizedtesting.SeedInfo.seed([2E3BF44EC3457CCA:DA6EA4532975FABD]:0) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.HitQueue.lambda$new$0(HitQueue.java:70) >>> at >>> org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.search.HitQueue$$Lambda/0x000071d2ec233150.get(Unknown >>> Source) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:105) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.HitQueue.<init>(HitQueue.java:63) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:203) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector.<init>(TopScoreDocCollector.java:50) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.TopScoreDocCollectorManager.newCollector(TopScoreDocCollectorManager.java:140) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.TopScoreDocCollectorManager.newCollector(TopScoreDocCollectorManager.java:28) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:741) >>> at org.apache.lucene.core@10.0.0-SNAPSHOT >>> /org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:725) >>> at >>> org.apache.lucene.sandbox.search.TestLargeNumHitsTopDocsCollector.testNoPQBuild(TestLargeNumHitsTopDocsCollector.java:111) >>> at >>> java.base/java.lang.invoke.LambdaForm$DMH/0x000071d2ec114000.invokeVirtual(LambdaForm$DMH) >>> at >>> java.base/java.lang.invoke.LambdaForm$MH/0x000071d2ec0bc800.invoke(LambdaForm$MH) >>> at >>> java.base/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) >>> at >>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:153) >>> at >>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) >>> at java.base/java.lang.reflect.Method.invoke(Method.java:580) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) >>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >>> /org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) >>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >>> /org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) >>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >>> /org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) >>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >>> /org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) >>> at org.apache.lucene.test_framework@10.0.0-SNAPSHOT >>> /org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) >>> at junit@4.13.1 >>> /org.junit.rules.RunRules.evaluate(RunRules.java:20) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) >>> at randomizedtesting.runner@2.8.1 >>> /com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: builds-h...@lucene.apache.org >> >>
