JingsongLi opened a new pull request, #1412:
URL: https://github.com/apache/paimon-rust/pull/1412

   ### Purpose
   
   Allow distributed column-backfill workers to execute complete Data Evolution 
file groups captured at one snapshot without replanning the table. The native 
shard updater previously always called `plan_all()`, while 
`TableUpdate.with_plan()` rejected shard operations. This is the Core 
prerequisite for switching PyPaimon's Ray workers to native reads and writes; 
Ray factory routing remains a separate PyPaimon change.
   
   ### Brief change log
   
   - Let `TableUpdate.with_plan()` feed shard updaters and expose the behavior 
through both batch and stream Python bindings.
   - Reuse row-ID mutation-plan validation and fresh worker-local 
authorization, preserving the captured snapshot as the conflict baseline.
   - Select balanced, complete logical file groups through `RowIdPlan`, 
retaining partial-column files, dedicated providers and deletion vectors. 
Workers with preassigned groups use shard `(0, 1)`.
   - Keep empty worker plans empty after concurrent appends, and continue 
rejecting selected/streaming mutation splits.
   
   ### Tests
   
   - `cargo test --locked -p paimon --all-targets --features vortex -- 
--test-threads=4`: 4,931 unit tests and 909 integration tests passed; 7 
ignored. Two existing subprocess/cache tests failed during the initial 
high-concurrency run, then the cache module's 24 tests and the complete 
lower-concurrency suite passed.
   - `make test-rust` in `bindings/python`: 151 passed.
   - `uv run --no-sync pytest tests/test_update.py -q` in `bindings/python`: 26 
passed.
   - Added regressions for manifest-free execution after concurrent appends, 
complete groups with partial-column files and deletion vectors, 
preassigned/noncontiguous groups, empty/oversharded plans, zero-column reads, 
commit conflicts, authorization changes and pickled worker splits.
   - Three review rounds found no actionable P0/P1/P2 findings.
   
   - `cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex 
-- -D warnings`: passed.
   - `cargo +1.98.0 fmt --all -- --check` and `git diff --check`: passed.
   
   ### API and Format
   
   Extend the existing `with_plan` contract to shard operations; no 
method-signature or storage/wire-format changes. Python workers transport the 
snapshot ID and picklable splits, reconstruct `Plan`, and use 
`with_plan(plan).new_shard_updator(0, 1)`. User functions continue to execute 
in Python.
   
   ### Documentation
   
   Updated the Python README and batch/stream type stubs with captured-plan 
execution and worker reconstruction.
   
   Implementation and review were AI-assisted.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to