binjie yang created SPARK-39023:
-----------------------------------
Summary: Add Executor Pod inter-pod anti-addinity
Key: SPARK-39023
URL: https://issues.apache.org/jira/browse/SPARK-39023
Project: Spark
Issue Type: New Feature
Components: Kubernetes
Affects Versions: 3.2.1
Reporter: binjie yang
h3. Why should we need this?
When Spark On Kubernetes is running, Executor Pod clusters occur in certain
conditions (uneven resource allocation in Kubernetes, high load On some nodes,
low load On some nodes), causing Shuffle data skew. This causes Spark
application to fail or performance bottlenecks, such as Shuffle Fetch timeout
and connection refuse after connection number.
h3. Why should use this?
The functionality mentioned in this PR was tested on a cluster.
Using three Kubernetes Node(node-1, node-2, node-3).
In the case of sufficient or insufficient cluster resources, it has the same
effect Whether the feature is enabled or not. Kubernetes assigns pods to nodes
with low load based on global resources. When only one node has a small load,
Kubernetes will schedule all executor pods to this node.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
