srowen commented on issue #23697: [SPARK-26758][core]Idle Executors are not getting killed after spark.dynamiAllocation.executorIdleTimeout value URL: https://github.com/apache/spark/pull/23697#issuecomment-459072285 Ah, I'm understanding better now. So, is the issue that an executor that has never gotten any task is never recorded in removeTimes, so is never removed? Looks like executors are marked as idle initially in onExecutorAdded, which ought to happen even before any task is scheduled. Hm, but start() allocates the initial executors, and I'd assume those result in calling onExecutorAdded(): ``` executorIds.filter(listener.isExecutorIdle).foreach(onExecutorIdle) ``` And onExecutorIdle() does initially mark it as idle and adds it to `removeTimes`. I wonder if the problem is somewhere else in there? maybe you know as you've looked into it more. I'm still not clear why it wouldn't get removed later, even if this fix does seem to improve the logic.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
