JingsongLi opened a new pull request, #10413: URL: https://github.com/apache/paimon/pull/10413
## Purpose Use Rust core for primary-key worker sharding instead of planning every bucket through Native and filtering the returned splits in Python. Depends on https://github.com/apache/paimon-rust/pull/1059. This PR is draft until that core change is available on Rust `main`; the Python CI Native job continues building Rust `main`. ## Changes - Pass the existing primary-key `with_shard` selection to Rust, including timestamp incremental windows. - Remove the second Python bucket filter. Java deletion-vector and first-row tables use file-name hash sharding, so a file owned by a worker can come from any bucket. - Let Rust apply ordinary primary-key LIMIT hints after selecting the shard. Keep distributed LIMIT deferred when Python still needs to refine sorted indexes, and retain the existing append row-position handling. - Correct the non-native primary-key planner to use Java `SnapshotReaderImpl.withShard` and `MergeTreeSplitGenerator.alwaysRawConvertible`: file-name hashes for DV/first-row and buckets for the other engines. No new Python or Rust public API and no new dependency. This only addresses existing PyPaimon scan capabilities. ## Validation - Added JDK 8 hash golden values, including UTF-16 supplementary characters, negative hashes, collisions and `Integer.MIN_VALUE`. - Real Parquet tests verify shard ownership, complete coverage without duplicates, deletion vectors, historical snapshots, incremental events and LIMIT. Guards reject Python planner/read fallback and reject Python LIMIT processing for ordinary Native PK shards. - Unit coverage retains LIMIT after Python sorted-index refinement and prevents a second bucket filter on Native file shards. - Tested with the companion Rust Python binding and `PYPAIMON_TEST_NATIVE_PLAN=1 PYPAIMON_TEST_NATIVE_READ=1 PYPAIMON_TEST_NATIVE_WRITE=1` across distribution, incremental, chunk shuffle, dynamic buckets, planner capabilities, expanded planning, batch level filters, split generation and adapter tests: **242 passed, 1 skipped, 108 subtests passed**; **502 Native plans, 495 Native reads, 454 Native writes**. - Changed Python files pass the repository Flake8 configuration and `git diff --check`. - Companion Rust core: 3745 library tests passed, 6 ignored; Clippy 1.98.0 and rustfmt passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
