JingsongLi commented on code in PR #8380:
URL: https://github.com/apache/paimon/pull/8380#discussion_r3497410974
##########
paimon-core/src/main/java/org/apache/paimon/deletionvectors/ApplyDeletionFileRecordIterator.java:
##########
@@ -67,7 +69,7 @@ public InternalRow next() throws IOException {
if (next == null) {
return null;
}
- if (!deletionVector.isDeleted(returnedPosition())) {
+ if (!deletionVector.isDeleted(offset + returnedPosition())) {
Review Comment:
A simpler way to preserve the existing iterator contract may be to shift the
DV before wrapping each file reader instead of passing `offset` into
`ApplyDeletionFileRecordIterator`. Since `returnedPosition()` is
file/reader-local, the anchor-range DV should be converted to reader-local
positions with `-offset` (`localPosition = anchorPosition - offset`), e.g. for
the 32-bit bitmap this can use `RoaringBitmap.addOffset(anchorBitmap, -offset)`
and ideally trim to the current reader range. Then
`ApplyDeletionFileRecordIterator` can keep checking
`deletionVector.isDeleted(returnedPosition())`, and consumers such as
`ArrowVectorizedBatchConverter` will also see the already-local, offset-aware
`DeletionVector` through `deletionVector()`. If bitmap64 is enabled, we should
apply the same shift when rebuilding the 64-bit DV as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
