BeiWeiDeDengDai opened a new issue, #2370: URL: https://github.com/apache/incubator-paimon/issues/2370
### Search before asking - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar. ### Motivation I logged and observed the consumption situation on ContinuousFileSplitEnumerator The first batch of data 15:38:29:5600w shards: 600 shards to be allocated 580 The second batch of data 15:43:19:5500w Partitions: 600 Partitions to be allocated 1118 The third batch of data 15:49:29 5500w Number of slices: 600 to be allocated 1652 ### Solution The reason is that there is backpressure in downstream processing. We hope to optimize the jobManager based on downstream processing speed and not add new shards ,Or there could be a better solution ### Anything else? _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
