JunRuiLee opened a new pull request, #11048: URL: https://github.com/apache/paimon/pull/11048
### Purpose Extend the distributed compaction recovery introduced in #9339 to Ray `update_by_row_id`, including calls made by a `process_row_id_ranges` processor. If compaction changes file boundaries after updates are staged, the current direct commit path relies on driver-local recovery and its affected-file size limit. Reuse the existing Ray coordinator to read staged updates on workers, reroute them against the latest file groups, and retry without rerunning source transformations. Support automatic partition sizing during recovery and preserve compatible schema evolution. Rebase-only checks run only when file boundaries require rewriting; written columns are resolved by historical field ID against the current schema. Nullable-column backfills, renames, and compatible type conversions remain supported, while conflicting logical updates and dropped update fields are rejected. The local recovery default remains **256 MB**. ### Tests Validated with Python 3.11, Ray 2.51.0, PyArrow 22.0.0, and DataFusion 54.1.0: - **77 tests and 8 subtests passed** across the row-ID update API, new compaction recovery cases, shared retry coordinator, conflict protocol, and eight existing self-merge recovery cases. - New cases cover range-batched processing with automatic/explicit parallelism, a `1 B` local recovery limit, no source replay, repeated compaction with a stable commit user, and preserved logical conflicts. - Schema regressions cover added-column backfills, schema changes after append without compaction, renamed fields with reused names and type widening, and dropped-field rejection. - Project-configured flake8, Python 3.6 syntax parsing for changed Python files, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
