dweiss opened a new pull request, #16566: URL: https://github.com/apache/lucene/pull/16566
This failure: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Linux/5940/console reproduces: ``` gradlew :lucene:core:test --tests "org.apache.lucene.index.TestIncrementalDocValuesUpdates.testSparseFoldOverDenseBase" -Ptests.asserts=false -Ptests.directory=MMapDirectory -Ptests.file.encoding=ISO-8859-1 -Ptests.gui=true -Ptests.haltonfailure=false -Ptests.jvmargs= -Ptests.jvms=6 -Ptests.multiplier=3 -Ptests.seed=1524CDD7264C1732 -Ptests.vectorsize=default ``` Here is the analysis from an LLM. I've added an explicit doc count instead of relyong on .cost in that test. --- testSparseFoldOverDenseBase runs with maxDocValuesOverlays=1 on a 10-doc segment. After the second full-corpus update round folds the column to a dense generation, it updates only d0 twice more and asserts the final fold stays a sparse overlay over the dense base. Instead the overlay comes back empty. The decision it's testing lives in ReadersAndUpdates.handleDVUpdates. When a fold is due (compact == true), the code estimates how much of the segment the prior delta generations cover by summing the iterator cost of each delta (ReadersAndUpdates.java:392-395): deltaCoverage += type == BINARY ? p.getBinary(fieldInfo).cost() : p.getNumeric(fieldInfo).cost(); and folds to a dense column — clearing the overlay — when deltaCoverage >= maxDoc * FOLD_TO_DENSE_COVERAGE_RATIO (0.5), at ReadersAndUpdates.java:406-407. The problem: SimpleText's numeric doc-values iterator reports cost() == maxDoc unconditionally (SimpleTextDocValuesReader.java:343-345 — the docsWithField iterator just returns maxDoc), no matter how many docs actually have a value in that generation. So the fourth update's fold sees a prior delta containing one doc (d0) but measures its coverage as 10; 10 ≥ 10 × 0.5, so foldToDense becomes true, the writer does a full-column rewrite, and clears the overlay (newOverlays.put(fieldInfo.number, new long[]{-1}) at line 575) — exactly the degradation the test is guarding against. Assessment DocIdSetIterator.cost() is documented as an estimate, and SimpleText's "cost = maxDoc" is a legal implementation of that contract (the comment at ReadersAndUpdates.java:376 even concedes the sum is >= distinct). Fold-to-dense is still value-correct, so this is a heuristic misfire, not data corruption — but it means the "stay sparse over a dense base" behavior silently never happens under any codec with pessimistic cost estimates, and the test's structural assertion only holds for codecs with exact costs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
