salvatorecampagna opened a new pull request, #16768: URL: https://github.com/apache/lucene/pull/16768
## TL;DR `DenseLiveDocs` and `SparseLiveDocs` accept a backing bit set longer than `maxDoc` but count it at its full length, so a bit set past `maxDoc` makes `deletedCount` wrong and makes two of the iterators return doc ids that don't exist. Both builders now require the backing set to be exactly `maxDoc` bits. ## Summary Both classes only require the backing set to be at least `maxDoc` long, while `length()` returns `maxDoc`. `applyMask` bounds on `maxDoc`, nothing else does, so a bit at or beyond `maxDoc` counts as a document. `deletedCount` comes out wrong in both directions, and `DenseLiveDocs.liveDocsIterator` and `SparseLiveDocs.deletedDocsIterator`, which wrap `BitSetIterator`, walk the backing set's own length and emit doc ids past `maxDoc`. `build` can't catch it, since it derives `deletedCount` from an unbounded `cardinality` and the result still lands inside `[0, maxDoc]`. I'd rather require exact length than bound every read. `Lucene90LiveDocsFormat` always passes a set of exactly `maxDoc`, and `length()` is the value passed in rather than something rounded to a word or block, so no caller in Lucene changes. Bounding would need a range `cardinality` on `SparseFixedBitSet`, which doesn't have one, plus bounded versions of those two iterators, all to keep an input shape nothing produces. > [!IMPORTANT] > This drops support for a padded backing set. `DenseLiveDocs.applyMask` documented it and one case in `testApplyMaskThrowsOnBitsBeyondLiveDocs` relied on it, so both go. The builders are public, so an out of tree caller passing a longer set now gets an `IllegalArgumentException`. Shipped in 10.5.0, so it applies to `branch_10x` too. Nothing in Lucene hits it today. ## Testing ``` ./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestLiveDocs" ./gradlew :lucene:core:test --tests "org.apache.lucene.util.TestFixedBitSet" --tests "org.apache.lucene.util.TestSparseFixedBitSet" ./gradlew :lucene:core:test --tests "org.apache.lucene.codecs.lucene90.TestLucene90LiveDocsFormat" ./gradlew :lucene:core:test --tests "org.apache.lucene.index.TestPendingDeletes" --tests "org.apache.lucene.index.TestPendingSoftDeletes" ./gradlew :lucene:core:test --tests "org.apache.lucene.index.TestSoftDeletesDirectoryReaderWrapper" --tests "org.apache.lucene.index.TestSoftDeletesRetentionMergePolicy" ./gradlew :lucene:core:test --tests "org.apache.lucene.index.TestIndexWriterDelete" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
