JunRuiLee opened a new pull request, #11048:
URL: https://github.com/apache/paimon/pull/11048

   ### Purpose
   
   Extend the distributed compaction recovery introduced in #9339 to Ray 
`update_by_row_id`, including calls made by a `process_row_id_ranges` 
processor. If compaction changes file boundaries after updates are staged, the 
current direct commit path relies on driver-local recovery and its 
affected-file size limit. Reuse the existing Ray coordinator to read staged 
updates on workers, reroute them against the latest file groups, and retry 
without rerunning source transformations.
   
   Support automatic partition sizing during recovery and preserve compatible 
schema evolution. Rebase-only checks run only when file boundaries require 
rewriting; written columns are resolved by historical field ID against the 
current schema. Nullable-column backfills, renames, and compatible type 
conversions remain supported, while conflicting logical updates and dropped 
update fields are rejected. The local recovery default remains **256 MB**.
   
   ### Tests
   
   Validated with Python 3.11, Ray 2.51.0, PyArrow 22.0.0, and DataFusion 
54.1.0:
   
   - **77 tests and 8 subtests passed** across the row-ID update API, new 
compaction recovery cases, shared retry coordinator, conflict protocol, and 
eight existing self-merge recovery cases.
   - New cases cover range-batched processing with automatic/explicit 
parallelism, a `1 B` local recovery limit, no source replay, repeated 
compaction with a stable commit user, and preserved logical conflicts.
   - Schema regressions cover added-column backfills, schema changes after 
append without compaction, renamed fields with reused names and type widening, 
and dropped-field rejection.
   - Project-configured flake8, Python 3.6 syntax parsing for changed Python 
files, and `git diff --check` passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to