[
https://issues.apache.org/jira/browse/FLINK-25648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuan Huang updated FLINK-25648:
--------------------------------
Description:
When creating Task Manager (TM) Pod, the fabric client needs the deployment to
set owner reference for the TM pod. However, repeating to querying the
deployment each time is unnecessary which will waste a lot of time.
https://github.com/apache/flink/blob/36ff71f5ff63a140acc634dd1d98b2bb47a76ba5/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
```
@Override
public CompletableFuture<Void> createTaskManagerPod(KubernetesPod
kubernetesPod) {
return CompletableFuture.runAsync(
() -> {
final Deployment masterDeployment =
this.internalClient
.apps()
.deployments()
.withName(KubernetesUtils.getDeploymentName(clusterId))
.get();
if (masterDeployment == null) {
throw new RuntimeException(
"Failed to find Deployment named "
+ clusterId
+ " in namespace "
+ this.namespace);
}
// Note that we should use the uid of the master Deployment for the
// OwnerReference.
setOwnerReference(
masterDeployment,
Collections.singletonList(kubernetesPod.getInternalResource()));
LOG.debug(
"Start to create pod with spec {}{}",
System.lineSeparator(),
KubernetesUtils.tryToGetPrettyPrintYaml(
kubernetesPod.getInternalResource()));
this.internalClient.pods().create(kubernetesPod.getInternalResource());
},
kubeClientExecutorService);
}
```
According to the measured result by
[https://arthas.aliyun.com/doc/trace.html#id1]) below: Quering deployment may
cost more than 50% (203/349 ms) of time.
Can we save the deployment reference in memory instead of querying it each time
to save pod creating time?
!trace.png!
was:
When creating Task Manager (TM) Pod, the fabric client needs the deployment to
set owner reference for the TM pod. However, repeating to querying the
deployment each time is unnecessary which will waste a lot of time.
According to the measured result by
https://arthas.aliyun.com/doc/trace.html#id1) below: Quering deployment may
cost more than 50% (203/349 ms) of time.
Can we save the deployment reference in memory instead of querying it each time
to save pod creating time?
!trace.png!
> Redundant to querying deployment when creating task manager pod
> ---------------------------------------------------------------
>
> Key: FLINK-25648
> URL: https://issues.apache.org/jira/browse/FLINK-25648
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
> Affects Versions: 1.14.2
> Reporter: Yuan Huang
> Priority: Critical
> Attachments: trace.png
>
>
> When creating Task Manager (TM) Pod, the fabric client needs the deployment
> to set owner reference for the TM pod. However, repeating to querying the
> deployment each time is unnecessary which will waste a lot of time.
>
> https://github.com/apache/flink/blob/36ff71f5ff63a140acc634dd1d98b2bb47a76ba5/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/Fabric8FlinkKubeClient.java
> ```
> @Override
> public CompletableFuture<Void> createTaskManagerPod(KubernetesPod
> kubernetesPod) {
> return CompletableFuture.runAsync(
> () -> {
> final Deployment masterDeployment =
> this.internalClient
> .apps()
> .deployments()
> .withName(KubernetesUtils.getDeploymentName(clusterId))
> .get();
> if (masterDeployment == null) {
> throw new RuntimeException(
> "Failed to find Deployment named "
> + clusterId
> + " in namespace "
> + this.namespace);
> }
> // Note that we should use the uid of the master Deployment for the
> // OwnerReference.
> setOwnerReference(
> masterDeployment,
> Collections.singletonList(kubernetesPod.getInternalResource()));
> LOG.debug(
> "Start to create pod with spec {}{}",
> System.lineSeparator(),
> KubernetesUtils.tryToGetPrettyPrintYaml(
> kubernetesPod.getInternalResource()));
> this.internalClient.pods().create(kubernetesPod.getInternalResource());
> },
> kubeClientExecutorService);
> }
> ```
> According to the measured result by
> [https://arthas.aliyun.com/doc/trace.html#id1]) below: Quering deployment may
> cost more than 50% (203/349 ms) of time.
> Can we save the deployment reference in memory instead of querying it each
> time to save pod creating time?
> !trace.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
