liupc edited a comment on issue #23697: [SPARK-26758][core]Idle Executors are not getting killed after spark.dynamiAllocation.executorIdleTimeout value URL: https://github.com/apache/spark/pull/23697#issuecomment-459189579 @sandeep-katta For my understanding, the executor allocation manager is to manage the executors according to the task we need, so if there are no stage submitted, why should it start to do real work(for instance, removing idle executors as you mentioned above)? Suppose we have set dynamic allocation target as minExecutor < initialExecutors < maxExecutors, then if initialExecutors is what we expected at starting, why we should idle executors? If we do the executor number may not be the initialExecutors when a stage is submitted, and it's very likely that the stage will need at least initialExecutors(for it's the initial target set by user), so the useless reallocation will happen. The reallocation may take some while and delay the overall stage execution time. So maybe I think what you describe is not a problem?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
