[
https://issues.apache.org/jira/browse/SPARK-36699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36699:
------------------------------------
Assignee: Apache Spark
> Reuse compatible executors for stage-level scheduling
> -----------------------------------------------------
>
> Key: SPARK-36699
> URL: https://issues.apache.org/jira/browse/SPARK-36699
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: Xiaochang Wu
> Assignee: Apache Spark
> Priority: Major
> Fix For: 3.3.0
>
>
> Current stage-level scheduling allocated separated set of executors for
> different profiles. This approach simplified implementation, however is a
> waste of executor resources when the existing executors have enough resources
> to run the following tasks.
> We proposed to reuse executors by defining a "compatible" executor concept:
> two executors binding to different resource profiles are compatible only when
> the executorResources (cores in particular if not defining custom resources)
> are the same, but taskResources can be different. When the executors are
> compatible, the tasks can be allocated to any of them even when in the
> different profiles. Users defining profiles should make sure the different
> taskResources are properly specified against the same executorResources.
> The typical user scenario is for different stages, user wants to use
> different core number for the task with same executor resources. For instance
> in CPU machine learning scenario, to achieve the best performance, given the
> same executor resources, when in ETL stage, user will allocate 1 core per
> task and many tasks, and in the following CPU training stage, user will use
> more cores per task and less tasks. Reusing executors allows better CPU
> resource utilization and better performance.
> The first PR will focus on reusing executors with same cores without custom
> resources. A SparkConf option is defined to change the default behavior which
> is not reusing executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
