XiaoHongbo-Hope commented on code in PR #11225:
URL: https://github.com/apache/paimon/pull/11225#discussion_r4608993892


##########
paimon-core/src/main/java/org/apache/paimon/operation/BlobFallbackRecordReader.java:
##########
@@ -120,6 +120,29 @@ public class BlobFallbackRecordReader implements 
RecordReader<InternalRow> {
                     .add(file);
         }
 
+        // Disjoint files cannot supply competing versions of the same row. 
Read them as a

Review Comment:
   > The all-disjoint fast path looks good and is safe since real rows keep 
their per-file sequence number and unresolved placeholders still fall through 
to `nullBlobRow(rowId)`.
   > 
   > One thought: because `disjoint` is currently an all-or-nothing check 
across all `files`, if a split has many disjoint base BLOB files (e.g., 100 
rolled BLOB files `[0, 999]@1`, `[1000, 1999]@2`, ..., `[99000, 99999]@100`) 
and even a **single** sub-range is later updated (e.g., `[0, 999]@101`), 
`disjoint` becomes `false` and the reader falls back to creating 101 
sequence-group readers instead of 2.
   > 
   > If we instead pack files in descending `maxSequenceNumber` order into the 
earliest group after the last group whose files overlap 
`file.nonNullRowIdRange()`, all-disjoint files still collapse into 1 group, 
while partially updated splits collapse into the minimum number of 
non-overlapping layers (e.g., 2 groups instead of 101) while preserving exact 
descending-sequence precedence for every row ID.
   
   Thanks for the suggestion!  I'd like to keep this PR focused on the 
all-disjoint case and follow up with a separate PR for layered grouping.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to