JingsongLi opened a new pull request, #838: URL: https://github.com/apache/paimon-rust/pull/838
### Purpose Fix the append shard mismatch exposed by the Rust Plan CI job in apache/paimon#9825. When manifest entries interleave partitions and buckets, scan planning currently groups all buckets of a partition together. Python instead preserves the first occurrence of each `(partition, bucket)` pair, so the same positional shard can select different rows. For example, groups encountered as `(b, 1), (a, 0), (b, 0), (b, 1)` must be emitted as `(b, 1), (a, 0), (b, 0)`, retaining both files in the first group. ### Brief change log - Replace nested partition/bucket maps with one insertion-ordered map keyed by `(partition, bucket)`. - Preserve file order within each group and its original total-bucket metadata. - Correct the grouping regression and add snapshot-planning coverage for interleaved groups with and without a pushed-down limit. ### Tests - The new snapshot-planning regression failed before the fix with the reordered groups. - `cargo test --locked -p paimon --lib table::table_scan::tests`: 101 passed, including append, primary-key, Data Evolution, row-position, range and limit tests. - `cargo clippy --locked -p paimon --lib --tests -- -D warnings` and `cargo fmt --all -- --check`: passed. - Built the Python wheel from this branch. In apache/paimon#9825, 345 related Python tests passed, including the original fixed-bucket append shard failure and a new comparison of all three shards, four slices, and shard/slice combinations with limits. The new comparison forbids native fallback and reads serially so limit results are deterministic. ### API and Format No public API or storage format changes. Scan groups retain manifest encounter order across partitions; this corrects the physical row positions consumed by Python append distribution. ### Documentation No documentation changes are required for this ordering fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
