gaobinlong commented on code in PR #16452:
URL: https://github.com/apache/lucene/pull/16452#discussion_r3795371721
##########
lucene/misc/src/java/org/apache/lucene/misc/search/MemoryAccountingBitsetCollectorManager.java:
##########
@@ -47,21 +55,25 @@ public MemoryAccountingBitsetCollector newCollector() {
@Override
public Result reduce(Collection<MemoryAccountingBitsetCollector> collectors)
{
- int globalMaxDocEnd = 0;
+ // Size the result to just cover the highest matched doc across all
collectors. Each
+ // collector's maxDocEnd is inflated by doSetNextReader to the full leaf
regardless of what
+ // actually matches, so keying off it can significantly over-allocate on
selective queries or
+ // narrow intra-segment slices; use the actual high-water mark tracked at
collect time.
+ int resultSize = 0;
for (MemoryAccountingBitsetCollector collector : collectors) {
- globalMaxDocEnd = Math.max(globalMaxDocEnd, collector.getMaxDocEnd());
+ int last = collector.getHighestSetBit();
+ if (last >= 0) {
+ resultSize = Math.max(resultSize, collector.getMinDocBase() + last +
1);
+ }
}
- // TODO: with intra-segment concurrency enabled, globalMaxDocEnd equals
the full index maxDoc
- // even when only a portion of the index was searched, causing
over-allocation of the result
- // bitset.
- FixedBitSet result = new FixedBitSet(globalMaxDocEnd);
+ FixedBitSet result = new FixedBitSet(resultSize);
Review Comment:
> Would it be okay to keep this PR scoped to the right-bound over-allocation
and open a followup for the left bound work?
That is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
