JunRuiLee opened a new pull request, #668: URL: https://github.com/apache/paimon-rust/pull/668
### Purpose Linked issue: N/A (draft; link an issue before marking ready for review) Primary-key merge-on-read currently feeds one input stream per data file into the LoserTree. On fragmented tables, peak merge fan-in therefore grows with file count even when many files have disjoint primary-key ranges. This change concatenates files with strictly non-overlapping key ranges into sorted runs, so peak LoserTree fan-in follows key-range overlap depth while preserving merge semantics. ### Brief change log - Decode manifest min/max keys with a composite-key comparator that matches physical row ordering, including unsigned binary-key ordering. - Partition files by overlapping key range and greedily pack each section into lazily concatenated sorted runs. - Preserve global merge behavior when a read spans multiple splits; safely fall back to one stream per file for missing, malformed, or inverted key ranges. - Apply the existing merge fan-in guard to planned merge inputs rather than raw file count. - Add the `read.merge-by-sorted-run` escape hatch, enabled by default and disableable with `false` for comparison or rollback. - Add regression coverage for integer, binary, and multi-column keys, overlap boundaries, invalid metadata, read equivalence, cross-split deduplication, fan-in limits, and Parquet budget behavior. ### Tests - `cargo fmt --all -- --check` - `cargo test -p paimon --lib` (2103 passed, 1 ignored) - `cargo test -p paimon table::merge_tree_split_generator::tests --lib` - `cargo test -p paimon table::kv_file_reader::tests --lib` - `cargo clippy -p paimon --lib --tests -- -D warnings` - `git diff --check` ### API and Format - Adds the table option `read.merge-by-sorted-run` with default `true`. - No storage-format or manifest-format change. ### Documentation - Documents `read.merge-by-sorted-run` in the SQL option reference. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
