Zakelly commented on PR #26640: URL: https://github.com/apache/flink/pull/26640#issuecomment-3026723961
> It will take longer end to end time for single TM to finish the state upload if reduce thread num from 4 to 2. It would cause checkpoint timeout for the largest job in worst case. I guess you want to generalize the solution for ForStStateBackend. If so, I agree to use a FLIP to discuss a more comprehensive solution, if you want to. But honestly, the speed control in ForStStateBackend is different with the jitter for rocksdb state backend as they will use different Flink configurations. If there is no side effect, shall we merge the diff first? Does a jitter slow down the end to end time of checkpoint? In current implementation, one takes the executor but do a sleep for a while, so eventually not all the threads are working at the same time. Thus what is the essential difference of current plan compared with shrinking the thread pool. I'm not proposing to do something with ForSt or any specified state backends. I'm suggesting controlling that in `CheckpointStreamFactory` and its `CheckpointStateOutputStream`, and let JM coordinate all the TM to achieve a overall QPS or BPS. That applies for all the state backend at one time. It is a huge problem that we should resolve, and I suggest not doing it in a fragmented way. It is not easy to use or maintain two mechanism, also retire one, so we'd better be cautious introducing one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
