JingsongLi commented on code in PR #9987:
URL: https://github.com/apache/paimon/pull/9987#discussion_r4068255700
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionSplitRead.java:
##########
@@ -250,11 +250,26 @@ private RecordReader<InternalRow> createReader(
} else {
suppliers.add(
() -> {
- if (skipByFileIndex(filters, needMergeFiles,
dataFilePathFactory)) {
+ if (skipByFileIndex(
+ filters,
+ needMergeFiles,
+ dataFilePathFactory,
+ rowRanges,
+ null)) {
return new EmptyFileRecordReader<>();
}
DeletionVectorWithRange deletionVector =
readDeletionVector(needMergeFiles,
deletionVectorFactory);
+ if (deletionVector != null
+ && !deletionVector.deletionVector.isEmpty()
+ && skipByFileIndex(
Review Comment:
[P2] Avoid evaluating the same file indexes twice for groups with a deletion
vector.
When the first `skipByFileIndex` call finds candidates, this call reopens
and reevaluates every applicable file index after loading a non-empty DV. If an
index is stored in a `.index` sidecar (the default in-manifest threshold is 500
B), each surviving merged group incurs another storage read per indexed file,
even when the DV deletes an unrelated row. Could we retain the first-pass index
results and intersect their bitmap candidates with the DV, checking an
all-deleted selection for `REMAIN` results? That would preserve the early
rejection without repeating index I/O.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
