xwu99 opened a new pull request #33941: URL: https://github.com/apache/spark/pull/33941
### What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-36699 We proposed to optionally change behavior of stage-level scheduling by reusing compatible executors. Two executors binding to different resource profiles are **compatible** only when the executorResources (cores in particular if not defining custom resources) are the same, but taskResources can be different. When the executors are compatible, the tasks can be allocated to any of them even when in the different profiles. Users defining profiles should make sure the different taskResources are properly specified against the same executorResources. A SparkConf option `spark.dynamicAllocation.reuseExecutors` is defined to change the default behavior which is not reusing executors. When this option is turned on, dynamic allocation will count all compatible executors number to meet init/min/max executor number restrictions. The first PR will focus on reusing executors with same cores without custom resources. ### Why are the changes needed? Current stage-level scheduling allocated separated set of executors for different executor profiles. This approach simplified implementation, however is a waste of executor resources when the existing executors have enough resources to run the following tasks. The typical user scenario is for different stages, user wants to use different core number for the task with same executor resources. For instance in CPU machine learning scenario, to achieve the best performance, given the same executor resources, when in ETL stage, user will allocate 1 core per task and many tasks, and in the following CPU training stage, user will use more cores per task and less tasks. In the existing implementation, two separated profiles and executors are created. Reusing executors will get better CPU resource utilization and better performance. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
