XiaoHongbo-Hope opened a new pull request, #11243:
URL: https://github.com/apache/paimon/pull/11243

   ### Purpose
   
   A Data Evolution descriptor scan can have one normal file spanning many 
disjoint BLOB files. Overlap-based grouping keeps the whole range in one split, 
leaving a single reader to process the BLOB files even when more source tasks 
are available.
   
   Add the opt-in `blob-descriptor.scan.max-files-per-split` option (default 
`0`, disabled). For a single normal file and disjoint BLOB ranges, create 
bounded `IndexedSplit` row ranges and keep only the relevant BLOB files in each 
split. Leading, internal and trailing gaps remain covered exactly once. This 
reuses the existing row-range reader without changing writes or persisted 
formats.
   
   Overlapping BLOBs, multiple normal files, deletion vectors, streaming, query 
authorization, existing index/range plans, limits and top-N retain the original 
planning path. Splitting can increase normal-file IO, so the option is 
explicitly enabled and tunable. This change is independent of #11225, which 
optimizes reader traversal rather than scan parallelism.
   
   ### Tests
   
   - Planner tests for range coverage, file pruning and unsupported-layout 
fallback.
   - Persisted-table test comparing split and unsplit descriptors, including 
null BLOBs and predicate pruning; checks limit and ordinary payload-read 
behavior.
   - Existing `DataEvolutionBatchScanTest` and `DataEvolutionSplitReadTest`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to