HuangZhenQiu commented on PR #26640: URL: https://github.com/apache/flink/pull/26640#issuecomment-3024031811
@Zakelly hanks for sharing your thoughts on this PR. The primary goal here is to address a bottleneck related to the remote object store. Interestingly, the issue isn't with upload speed, but rather with the metadata database (MDDB) used to fetch tokens for each new file. In our largest job, we have around 1,000 TaskManagers (TMs), and each TM attempts to upload roughly 50 state files simultaneously. This creates a significant QPS load on the MDDB for a single bucket. As a result, not only does this lead to checkpoint failures in the affected job, but it also causes interference with other jobs — effectively acting as a noisy neighbor. After apply jitter configured with 10 seconds, we find the QPS to MDDB reduced an order of magnitude. Given this scenario, I believe introducing jitter is a more reasonable approach. It allows us to mitigate the MDDB pressure without sacrificing upload performance by reducing the number of upload threads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
