XiaoHongbo-Hope opened a new pull request, #9033: URL: https://github.com/apache/paimon/pull/9033
### Purpose Allow long-running Ray primary-key updates to commit periodically and resume after a driver restart. ### Changes - add `PaimonOffsetSource` with a pinned source snapshot and stable offsets; - add `commit_mode="incremental"`, `operation_id`, `commit_interval_seconds`, and `update_cols` to `write_paimon`; - store source progress in the same target snapshot as each data commit; - retain the source snapshot and latest checkpoint with internal tags; - reject concurrent target commits and schema changes; - add `delete_write_paimon_checkpoint` for completed operations. The default atomic path is unchanged. Incremental mode currently supports fixed-bucket primary-key targets using `merge-engine=partial-update`; sequence fields and cross-partition updates are not supported. ### Tests - real Ray E2E: time-triggered commits, restart recovery, empty results, inserts, concurrent commits, and schema changes; - existing Ray write and commit regression suites; - project flake8 and `git diff --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
