zhuyaogai opened a new pull request, #9247: URL: https://github.com/apache/paimon/pull/9247
### Purpose Fixes #7155. Previously, orphan file cleanup for all matched tables was unioned into a single Flink batch job. In batch mode, Flink does not put all job vertices into the same SlotSharingGroup by default, so tasks belonging to different groups cannot reuse the same slots. As each table creates its own cleanup pipeline, running many tables in one job can require a large number of slots. Therefore, the actual resource usage may be much higher than the configured cleanup parallelism and can exhaust cluster resources. This change processes tables in sequential batches and introduces a `table_batch_size` option, which defaults to 10. Only a limited number of table cleanup pipelines are included in each batch, bounding the slot resources required by a single job. Batch progress and elapsed time are also logged. ### Tests Added integration test coverage for database-level orphan file cleanup with a configured batch size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
