[
https://issues.apache.org/jira/browse/FLINK-33880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801645#comment-17801645
]
Yuan Huang commented on FLINK-33880:
-------------------------------------
Hi [~guoyangze] , How do you think this issue? Is it necessary to fix this
problem on the Flink side? I have prepared the fix in my local branch. If you
think this fix is necessary, can you assign this ticket to me and let me push
the Pull Request to the Flink repo?
> Introducing Retry Mechanism for Listing TaskManager Pods to Prevent API
> Server Connection Failures
> --------------------------------------------------------------------------------------------------
>
> Key: FLINK-33880
> URL: https://issues.apache.org/jira/browse/FLINK-33880
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
> Affects Versions: 1.17.2
> Reporter: Yuan Huang
> Priority: Major
> Attachments: image-2023-12-19-18-41-41-308.png,
> image-2023-12-19-18-44-13-623.png, image-2023-12-21-10-12-37-667.png
>
>
> When operating in Kubernetes mode, if the JobManager undergoes a restart, it
> attempts to establish a connection with the API server to retrieve the
> complete list of TaskManager Pods, facilitating the recovery of previous
> TaskManagers.
> In the context of a large Kubernetes cluster with potentially thousands of
> concurrently running jobs, a scenario may arise where all JobManagers undergo
> a restart and subsequently connect to the API server (e.g., during disaster
> recovery). This influx of requests may overwhelm the API server, reaching its
> maximum capacity and leading to the refusal of some JobManager requests.
> Consequently, certain JobManagers may experience failures and initiate
> reconnection attempts to the API server.
> !image-2023-12-21-10-12-37-667.png|width=609,height=305!
> !image-2023-12-19-18-44-13-623.png|width=505,height=206!
> To enhance this process, we can propose the implementation of a retry
> mechanism. In the event of a failed connection attempt to the API server,
> Flink will introduce a waiting period before making subsequent connection
> attempts, mitigating the risk of overwhelming the server and improving the
> overall resilience of the system.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
