holdenk commented on issue #23401: [SPARK-26513][Core] : Trigger GC on executor node idle URL: https://github.com/apache/spark/pull/23401#issuecomment-450700797 For "Dynamic allocation probably eliminates most of the benefit because of executor turn-over" I think we could still see some benefits. Dynamic scale down is often done fairly conservatively when combined with cached blocks, see the separate `spark.dynamicAllocation.cachedExecutorIdleTimeout` setting default of infinity (although in practice folks will likely choose a lower value) -- although if we start migrating cached blocks in the future that might be less useful. Even without the cached executor idle timeout, I could see folks having the executor timeout for decommissioning being longer than the amount of time it takes to run a GC event. Especially if we consider the interactive notebook case.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
