XiaoHongbo-Hope opened a new pull request, #9056: URL: https://github.com/apache/paimon/pull/9056
### Purpose Allow long-running Ray Dataset partial updates to commit completed work periodically. ### Changes - add `commit_mode="incremental"`, `commit_interval_seconds`, and `update_cols` to `write_paimon`; - commit fixed-bucket primary-key updates in time-based windows; - keep successful groups visible after failure and abort uncommitted files; - reject concurrent data or schema changes while allowing compaction. The target must use `merge-engine=partial-update`. This PR does not checkpoint source progress; retrying reruns the Ray Dataset. ### Tests - real Ray E2E for periodic commits, failure handling, 4096 buckets, partitioned composite keys, compaction, and concurrent changes; - Ray sink and commit regression tests; - project flake8 and `git diff --check`. Validation: 48 passed, 10 subtests passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
