xiangfu0 commented on code in PR #19039:
URL: https://github.com/apache/pinot/pull/19039#discussion_r3632266628
##########
pinot-core/src/main/java/org/apache/pinot/core/query/pruner/SelectionQuerySegmentPruner.java:
##########
@@ -97,8 +97,12 @@ public List<IndexSegment> prune(List<IndexSegment> segments,
QueryContext query)
return List.of(segments.get(0));
}
- // Skip pruning segments for upsert table because valid doc index is
equivalent to a filter
- if (segments.get(0).getValidDocIds() != null) {
+ // Skip pruning when a post-selection doc mask makes the raw total-doc
count overstate the surviving rows, which
+ // would otherwise under-return under LIMIT. Both signals are set
uniformly across a table's segments, so the
+ // first segment suffices: upsert's valid-doc index, or an
externally-supplied deleted-doc set. Skipping keeps
+ // results correct -- every segment is still scanned and the mask applied
during the scan.
+ IndexSegment firstSegment = segments.get(0);
+ if (firstSegment.getValidDocIds() != null ||
firstSegment.hasDeletedDocIds()) {
Review Comment:
Checking only the first segment leaves the wrong-result bug in place when
deleted-doc masks are segment-specific. `hasDeletedDocIds()` is a
per-`IndexSegment` flag, defaults to false, and is set on individual
`ImmutableSegmentImpl` instances; nothing enforces the asserted table-wide
uniformity. For `LIMIT 15`, segments with live/raw counts A=10/10, B=1/10
(flagged), and C=10/10, with A first, will keep A+B because their raw count
reaches 20 and prune C, returning only 11 rows. Check every segment for either
mask before enabling this pruning, and add a mixed-flag regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
