wankunde opened a new pull request, #44981: URL: https://github.com/apache/spark/pull/44981
### What changes were proposed in this pull request? In https://issues.apache.org/jira/browse/SPARK-13723 , conf --num-executors will be used to compute initialExecutors and initial # of executors. I think that may conflict to DYN_ALLOCATION_MIN_EXECUTORS and will not free the idle executors. ``` private def removeExecutors(executors: Seq[(String, Int)]): Seq[String] = synchronized { if (newExecutorTotal - 1 < minNumExecutors) { logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " + s"are only $newExecutorTotal executor(s) left (minimum number of executor limit " + s"$minNumExecutors)") } else if (newExecutorTotal - 1 < numExecutorsTargetPerResourceProfileId(rpId)) { logDebug(s"Not removing idle executor $executorIdToBeRemoved because there " + s"are only $newExecutorTotal executor(s) left (number of executor " + s"target ${numExecutorsTargetPerResourceProfileId(rpId)})") } else { executorIdsToBeRemoved += executorIdToBeRemoved numExecutorsTotalPerRpId(rpId) -= 1 } } ``` ### Why are the changes needed? Remove the idle executors and improve the cluster resources utilization. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Local test. Before this PR: ``` 24/01/27 02:41:20 INFO Utils: Using initial executors = 1000, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances ``` After this PR: ``` 24/01/27 02:51:20 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors and spark.dynamicAllocation.minExecutors ``` ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
