Zouxxyy commented on PR #9017: URL: https://github.com/apache/paimon/pull/9017#issuecomment-5176728648
Thanks. Foreground rescale targets a specific case: a small historical write created only one or a few buckets, but the new batch is very large. Without rescale, the fixed write would inherit that small bucket count and funnel the large batch into one or a few tasks, causing severe skew or OOM. We therefore stage and measure the new batch first, and rescale the small historical layout only when the required bucket number exceeds the existing one by 32x by default. This keeps the rewrite cost small while unlocking enough parallelism for the large new batch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
