Sasilekha commented on code in PR #16452:
URL: https://github.com/apache/lucene/pull/16452#discussion_r3794272289
##########
lucene/misc/src/java/org/apache/lucene/misc/search/MemoryAccountingBitsetCollectorManager.java:
##########
@@ -47,21 +55,25 @@ public MemoryAccountingBitsetCollector newCollector() {
@Override
public Result reduce(Collection<MemoryAccountingBitsetCollector> collectors)
{
- int globalMaxDocEnd = 0;
+ // Size the result to just cover the highest matched doc across all
collectors. Each
+ // collector's maxDocEnd is inflated by doSetNextReader to the full leaf
regardless of what
+ // actually matches, so keying off it can significantly over-allocate on
selective queries or
+ // narrow intra-segment slices; use the actual high-water mark tracked at
collect time.
+ int resultSize = 0;
for (MemoryAccountingBitsetCollector collector : collectors) {
- globalMaxDocEnd = Math.max(globalMaxDocEnd, collector.getMaxDocEnd());
+ int last = collector.getHighestSetBit();
+ if (last >= 0) {
+ resultSize = Math.max(resultSize, collector.getMinDocBase() + last +
1);
+ }
}
- // TODO: with intra-segment concurrency enabled, globalMaxDocEnd equals
the full index maxDoc
- // even when only a portion of the index was searched, causing
over-allocation of the result
- // bitset.
- FixedBitSet result = new FixedBitSet(globalMaxDocEnd);
+ FixedBitSet result = new FixedBitSet(resultSize);
Review Comment:
Thanks @gaobinlong, good catch on the left-bound waste
Looking at it though tightening the left bound would need a small API
change. Result.bitset today is indexed by absolute docId so a tighter bitset
would need either an offset field on Result(record signature changes) or
different return type from bitSet() a wrapper or SparseFixedBitSet). Either
way it feels worth designing on its own rather than folding into this bug fix
Would it be okay to keep this PR scoped to the right-bound over-allocation
and open a follow-up for the left-bound work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
