Zakelly commented on PR #26640: URL: https://github.com/apache/flink/pull/26640#issuecomment-3026574112
@HuangZhenQiu Thanks for your answer. I see your point about the metadata controller (e.g. NameNode) being the bottleneck. However this is still under the speed control story, given QPS/RPS and BPS are two aspects of the limit. This essentially is the QPS by jitter is lower than the case without jitter, mitigating the pressure on NN, but also slower the checkpoint. This is kind of sacrificing the upload performance, and I believe it can also be achieved effectively by shrinking the thread pool. 4 threads to 2 cuts down the QPS by half for example. Did you try this in your environment? I'm not doubting the value of jitter, it is a nice mechanism for backoff when some limit is reached or waiting for retry. It can mitigate the issue at hand but not the most suitable one. So I'd prefer holding this PR and considering a jitter in the comprehensive speed control design. I will kickoff a FLIP discussion about the design I proposed above, targeting next release. And we may cooperate on that if you are interested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
