JingsongLi opened a new pull request, #1059: URL: https://github.com/apache/paimon-rust/pull/1059
## Purpose PyPaimon already supports distributed primary-key scans, but Rust currently rejects `with_shard` unless chunk shuffling is enabled. Python therefore plans every bucket through Native, filters the returned splits itself, and applies the distributed LIMIT hint afterwards. Enable ordinary worker sharding in Rust core using Java's `SnapshotReaderImpl.withShard` and `SplitGenerator.alwaysRawConvertible` rules: - Append tables and primary-key tables with deletion vectors or the first-row merge engine select files using `abs(String.hashCode(fileName) % count)`. - Other primary-key tables and data-evolution tables select buckets using `bucket % count`, retaining complete merge-key and column groups. - Filter manifest entries before grouping, split packing and LIMIT pushdown. Snapshot selection and combined APPEND-delta planning share this path. - Keep chunk shuffle's existing balanced chunk distribution and row-position selection's separate API. This extends the existing API without adding dependencies. The Python binding remains a wrapper around core planning. Java references: [SnapshotReaderImpl](https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java), [MergeTreeSplitGenerator](https://github.com/apache/paimon/blob/master/paimon-core/src/main/java/org/apache/paimon/table/source/MergeTreeSplitGenerator.java). ## Validation - Added regression tests for Java UTF-16 hashes, negative hashes and `Integer.MIN_VALUE`, worker ownership, LIMIT ordering, empty shards, time travel, combined deltas, all merge engines and data-evolution column groups. - Verified the golden hash fixtures with JDK 8 `String.hashCode` and `Math.abs(hash % count)`. - `cargo test -p paimon --lib`: 3745 passed, 6 ignored. - `cargo +1.98.0 clippy -p paimon --all-targets -- -D warnings`: passed. - `cargo +1.98.0 fmt --all -- --check` and `git diff --check`: passed. - Built the Python binding and exercised the companion PyPaimon integration tests against it, including real Parquet reads and deletion vectors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
