JingsongLi opened a new pull request, #1099: URL: https://github.com/apache/paimon-rust/pull/1099
## Purpose Move PyPaimon's existing shard column-update operation into Rust core so Native mode owns the scan, whole-file sharding, visible row-ID mapping and partial-file staging. This adds no new Python operation or format. ## Changes - Add `TableUpdate.with_read_projection`, `new_shard_updater` and `ShardTableUpdater` in core; keep the Python-facing `new_shard_updator` spelling used by PyPaimon. - Capture one complete snapshot and balance complete normal-file row-ID groups. Column files overlapping an anchor stay together, and dedicated files do not redefine the groups. - Record the actual visible row IDs while reading, including deletion-vector filtering, and associate replacement batches in reader order. Input and output batch boundaries may differ. - Buffer one file group at a time and reuse `DataEvolutionWriter` and the shared input conversion rules. Require reader completion and exactly one replacement row per visible row before returning commit messages. - Keep failures terminal, including cancellation while a consuming file preparation is awaiting I/O. EOF cannot reset a failure; corrected input validation errors can be retried before rows are consumed. - Preserve prepared files on close and return identical messages for repeated preparation. Returned readers can outlive the updater object. - Add thin batch/stream Python bindings, type stubs and documentation. ## Java alignment The public shard API follows an existing PyPaimon operation. File writing reuses core's Data Evolution implementation: partial `write_cols`, original physical row-ID ranges, deletion-vector handling and `check_from_snapshot`. Java `BaseAppendFileStoreWrite.withWriteType` defines the column metadata rules; `FileStoreCommitImpl.configureRowIdCheckFromMessages` defines the captured snapshot conflict check. There is no new Python-specific file layout. ## Validation - 10 focused core tests pass, including actual storage-I/O cancellation, failure followed by EOF, partitioned/sharded updates, deleted rows, input correction and prepared-file ownership. - `cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets -- -D warnings` passes. - `cargo fmt --all --check` and `git diff --check` pass. - Rebuilt the Python extension and ran 527 PyPaimon tests with all five Native CI flags: all pass. Native shard update calls: 56; all existing update categories also exercised. - Independent core and wrapper review findings were fixed and verified. A companion Python PR routes the existing shard updater through these bindings and adds REST integration coverage. Python's Native CI will continue using Rust `main` once this PR is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
