[
https://issues.apache.org/jira/browse/FLINK-34728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828263#comment-17828263
]
Fei Feng commented on FLINK-34728:
----------------------------------
Yes, this is not an issue on the Flink kubernetes operator side, the operator
is just a client, and the server side needs to provide new api in order to
complete this lightweight submission remodeling.
> operator does not need to upload and download the jar when deploying session
> job
> --------------------------------------------------------------------------------
>
> Key: FLINK-34728
> URL: https://issues.apache.org/jira/browse/FLINK-34728
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Kubernetes Operator
> Affects Versions: kubernetes-operator-1.5.0, kubernetes-operator-1.6.0,
> kubernetes-operator-1.7.0
> Reporter: Fei Feng
> Priority: Major
> Attachments: image-2024-03-19-15-59-20-933.png
>
>
> Problem:
> By reading the source code of the sessionjob's first reconcilition in the
> session mode of the flink kubernetes operator, a clear single point of
> bottleneck can be identified. When submitting a session job, the operator
> needs to first download the job jar from the jarURL to the local storage of
> kubernetes pod , then upload the jar to the job manager through the
> `/jars/upload` rest api, and finally call the `/jars/:jarid/run` rest api to
> launch the job.
> In this process, the operator needs to first download the jar and then upload
> the jar. When multiple jobs are submitted to the session cluster
> simultaneously, the operator can become a single point of bottleneck, which
> may be limited by the network traffic or other resource constraints of the
> operator pod.
>
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L824]
> !image-2024-03-19-15-59-20-933.png|width=548,height=432!
>
> Solution:
> We can modify the job submission process in the session mode. The jobmanager
> can provide a `/jars/run` rest api that supports self-downloading the job
> jar, and the operator only needs to send a rest request to submit the job,
> without download and upload the job jar. In this way, the submission pressure
> of the operator can be distributed to each job manager.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
