XiaoHongbo-Hope commented on code in PR #11225:
URL: https://github.com/apache/paimon/pull/11225#discussion_r4608993892
##########
paimon-core/src/main/java/org/apache/paimon/operation/BlobFallbackRecordReader.java:
##########
@@ -120,6 +120,29 @@ public class BlobFallbackRecordReader implements
RecordReader<InternalRow> {
.add(file);
}
+ // Disjoint files cannot supply competing versions of the same row.
Read them as a
Review Comment:
> The all-disjoint fast path looks good and is safe since real rows keep
their per-file sequence number and unresolved placeholders still fall through
to `nullBlobRow(rowId)`.
>
> One thought: because `disjoint` is currently an all-or-nothing check
across all `files`, if a split has many disjoint base BLOB files (e.g., 100
rolled BLOB files `[0, 999]@1`, `[1000, 1999]@2`, ..., `[99000, 99999]@100`)
and even a **single** sub-range is later updated (e.g., `[0, 999]@101`),
`disjoint` becomes `false` and the reader falls back to creating 101
sequence-group readers instead of 2.
>
> If we instead pack files in descending `maxSequenceNumber` order into the
earliest group after the last group whose files overlap
`file.nonNullRowIdRange()`, all-disjoint files still collapse into 1 group,
while partially updated splits collapse into the minimum number of
non-overlapping layers (e.g., 2 groups instead of 101) while preserving exact
descending-sequence precedence for every row ID.
Thanks for the suggestion! I'd like to keep this PR focused on the
all-disjoint case and follow up with a separate PR for layered grouping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]