dwangatt commented on PR #8583: URL: https://github.com/apache/paimon/pull/8583#issuecomment-4978923485
> When `size-aware-batch` is enabled and `scan.split-enumerator.mode=preemptive`, the source ignores the bucket grouping function, but the sink skips the bucket shuffle. Multiple splits from the same bucket may end up in different writers, resulting in concurrent full compaction. It is recommended to enforce or validate `FAIR`, or to implement grouping and allocation for `PREEMPTIVE`. @JingsongLi thanks for review. Good point! I updated this PR - size-aware-batch option only works with `FAIR` mode. `size-aware-batch` requires all writers ready to eliminate long tail writers. In preemptive mode, current `LINEAR` can work well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
