I pushed a fix for this case as well as the other recent fail, which was due to an incorrect assertion added in a recent change on the random-access-vectors branch that was adding safety to the iterator used during merging, where we assume that only a single iterator will be used to access values at once. The previous assertion assumed that all vector values would be consumed, but during quantization we sample the vectors, iterating over all of them, but only retrieving the values for some, so the new safety check keeps track of the ords iterated over rather than the ords whose values are retrieved.
On Sat, Sep 28, 2024 at 5:15 PM Michael Sokolov <msoko...@gmail.com> wrote: > > These failures relate to the way Arrays.binarySearch works when there > are repeated values, in which case the result is undefined (it can be > any of the indexes with the value), but in SlowCompositeReaderWrapper > we are relying on finding the lowest-indexed of the repeats. I'll work > on a fix > > On Sat, Sep 28, 2024 at 3:32 PM Policeman Jenkins Server > <jenk...@thetaphi.de> wrote: > > > > Build: https://jenkins.thetaphi.de/job/Lucene-main-Linux/50364/ > > Java: 64bit/hotspot/jdk-24-ea+16 -XX:-UseCompressedOops -XX:+UseParallelGC > > > > 2 tests failed. > > FAILED: > > org.apache.lucene.backward_codecs.lucene90.TestLucene90HnswVectorsFormat.testSparseVectors > > > > Error Message: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > > > Stack Trace: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:916) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:929) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1550) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1838) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1478) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:224) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.index.BaseKnnVectorsFormatTestCase.testSparseVectors(BaseKnnVectorsFormatTestCase.java:873) > > at > > java.base@24-ea/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > > at java.base@24-ea/java.lang.reflect.Method.invoke(Method.java:573) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/junit@4.13.1/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) > > at > > app/junit@4.13.1/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) > > at java.base@24-ea/java.lang.Thread.run(Thread.java:1576) > > Caused by: java.lang.NullPointerException: Cannot invoke > > "org.apache.lucene.index.FloatVectorValues.vectorValue(int)" because > > "this.subs[this.lastSubIndex].sub" is null > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SlowCompositeCodecReaderWrapper$SlowCompositeKnnVectorsReaderWrapper$MergedFloatVectorValues.vectorValue(SlowCompositeCodecReaderWrapper.java:905) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SortingCodecReader$SortingFloatVectorValues.vectorValue(SortingCodecReader.java:324) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.KnnVectorsWriter$MergedVectorValues$MergedFloat32VectorValues.vectorValue(KnnVectorsWriter.java:364) > > at > > org.apache.lucene.backward_codecs.lucene90.Lucene90HnswVectorsWriter.writeVectorData(Lucene90HnswVectorsWriter.java:194) > > at > > org.apache.lucene.backward_codecs.lucene90.Lucene90HnswVectorsWriter.writeField(Lucene90HnswVectorsWriter.java:122) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.BufferingKnnVectorsWriter.mergeOneField(BufferingKnnVectorsWriter.java:203) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:121) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:158) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5283) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4749) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6567) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729) > > > > > > FAILED: > > org.apache.lucene.backward_codecs.lucene92.TestLucene92HnswVectorsFormat.testSparseVectors > > > > Error Message: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > > > Stack Trace: > > org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:916) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:929) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1550) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1838) > > at > > app/org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1478) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:224) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.index.BaseKnnVectorsFormatTestCase.testSparseVectors(BaseKnnVectorsFormatTestCase.java:873) > > at > > java.base@24-ea/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) > > at java.base@24-ea/java.lang.reflect.Method.invoke(Method.java:573) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/junit@4.13.1/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) > > at > > app/org.apache.lucene.test_framework@10.0.0-SNAPSHOT/org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) > > at > > app/junit@4.13.1/org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) > > at > > app/randomizedtesting.runner@2.8.1/com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) > > at java.base@24-ea/java.lang.Thread.run(Thread.java:1576) > > Caused by: java.lang.NullPointerException: Cannot invoke > > "org.apache.lucene.index.FloatVectorValues.vectorValue(int)" because > > "this.subs[this.lastSubIndex].sub" is null > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SlowCompositeCodecReaderWrapper$SlowCompositeKnnVectorsReaderWrapper$MergedFloatVectorValues.vectorValue(SlowCompositeCodecReaderWrapper.java:905) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SortingCodecReader$SortingFloatVectorValues.vectorValue(SortingCodecReader.java:324) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.KnnVectorsWriter$MergedVectorValues$MergedFloat32VectorValues.vectorValue(KnnVectorsWriter.java:364) > > at > > org.apache.lucene.backward_codecs.lucene92.Lucene92HnswVectorsWriter.writeVectorData(Lucene92HnswVectorsWriter.java:197) > > at > > org.apache.lucene.backward_codecs.lucene92.Lucene92HnswVectorsWriter.writeField(Lucene92HnswVectorsWriter.java:129) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.BufferingKnnVectorsWriter.mergeOneField(BufferingKnnVectorsWriter.java:203) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.mergeOneField(PerFieldKnnVectorsFormat.java:121) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.codecs.KnnVectorsWriter.merge(KnnVectorsWriter.java:105) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SegmentMerger.mergeVectorValues(SegmentMerger.java:271) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:158) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5283) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4749) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6567) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:668) > > at > > org.apache.lucene.core@10.0.0-SNAPSHOT/org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:729) > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org > > For additional commands, e-mail: builds-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
