dwangatt opened a new pull request, #9368: URL: https://github.com/apache/paimon/pull/9368
### Purpose The existing FAIR split assignment uses row count as the split weight. This works well when records have similar sizes, but can create significant workload skew for tables containing records with highly variable payload sizes. Two splits with the same row count may represent very different amounts of data. As a result, splits containing large-payload records can become concentrated on a small number of source subtasks or TaskManagers, causing: • Uneven S3 read and network load • Higher memory and CPU pressure on a subset of TaskManagers • Long-running straggler tasks • Longer overall batch and rescale job duration This change introduces a new split weight mode: scan.split-enumerator.weight-mode=file-size When used with the FAIR split assigner, Paimon calculates each DataSplit weight from the total size of its data files rather than its row count. The assigner can therefore distribute byte-heavy splits more evenly across source subtasks. For split types without file-size information, the implementation falls back to row-count weighting. The goal is to improve source workload balancing for large-payload tables—particularly during Paimon rescale and other bounded batch reads—by spreading heavy IO tasks across more TaskManagers and reducing execution long tails. ### Tests Add some tests to cover new code. Also tested in our FLINK batch job and new option reduced job running time from 8 hours -> 2 hours. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
