XiaoHongbo-Hope opened a new pull request, #9335: URL: https://github.com/apache/paimon/pull/9335
## Purpose Ray `_ROW_ID` self-merge already avoids the source-target join, but its update path still sends every transformed row through a `_FIRST_ROW_ID` hash shuffle before writing. This adds avoidable network, object-store, and scheduling overhead for large column updates. ## Changes - pin the target scan plan used by self-merge updates - dispatch each complete planned split directly to a bounded Ray task - read, apply matched clauses, and stage row-id updates within that task - collect normal `CommitMessage`s and commit once on the driver - stop scheduling new work after a task failure and abort files staged by completed tasks The public API is unchanged. General source-target merge and self-merge delete paths are unchanged. This PR is based on `master` and does not depend on #9327; callable self-merge can reuse this execution path after rebasing. ## Tests - self-merge updates succeed when `Dataset.groupby()` is disabled - a later split failure aborts files staged by earlier splits - `ray_data_evolution_merge_into_test.py`: 109 passed - `ray_update_by_row_id_test.py`: 16 passed - flake8, compileall, and `git diff --check` passed Local microbenchmark (Ray 2.56.1, 4 CPUs, 50,000 rows, 10 files, 1 KiB binary payload): 3.49 s with the routing shuffle and 1.11 s without it. This is a local directional result, not a production-table claim. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
