QuakeWang opened a new pull request, #8541:
URL: https://github.com/apache/paimon/pull/8541
### Purpose
Ray `map_groups` only keeps a `(partition, bucket)` group intact while the
UDF runs. Its output may be split before `PaimonDatasink`, so sink options that
prevent operator fusion can create multiple independent writers for one bucket
with overlapping sequence numbers.
This change writes each HASH_FIXED primary-key group inside the `map_groups`
UDF, returns serialized commit messages, and commits them on the driver. Writer
concurrency and remote options are applied directly to the group tasks.
### Tests
- Added regression coverage for output block splitting and non-fused
writer options.
- Added grouped overwrite and empty-overwrite coverage.
- Passed Ray 2.54.0 and 2.56.0 related tests, full Python flake8, license
checks, and docs build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
