Guosmilesmile opened a new pull request, #14837: URL: https://github.com/apache/iceberg/pull/14837
When using Flink to rewrite data files, we already set a max file group size, but we still encounter uneven file counts after planning: groups 1–100 each have about 50 files, while group 50 contains 5,000 files. In Flink this causes severe backpressure: all other slots become idle and wait until group 101 finishes,and new groups can not be assigned to the idle slot. This leads to wasted resources and lengthens the compaction job, which in turn increases the number of small files. This PR mainly adds a max file group count parameter in the planner. For the scenario above, it can split the imbalanced groups into smaller groups to alleviate Flink backpressure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
