Bowen created SPARK-50277:
-----------------------------
Summary: [k8s] Apply for executor pods in parallel
Key: SPARK-50277
URL: https://issues.apache.org/jira/browse/SPARK-50277
Project: Spark
Issue Type: Improvement
Components: k8s, Kubernetes
Affects Versions: 3.5.1
Reporter: Bowen
The performance of spark on k8s is worse than that of yarn. It is found that
the application of executor pod is executed sequentially. The k8s interface for
applying pod is
kubernetesClient.pods().inNamespace(namespace).resource(podWithAttachedContainer).create(),
which is asynchronous. However, each execution still takes an average of
62.57ms. Applying 280 pods takes 17520ms, which means that the speed of
applying pod is about 15-16 pods/second. If a job requires more executors, this
speed will become a bottleneck. I would like to ask whether this logic can be
changed to concurrently apply for executor pods, and whether there will be any
negative impact.
The logic of applying for executor is in method：
org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator#requestNewExecutors
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
