Github user tgravescs commented on the issue:
    I'm saying you have a stage running that has > 0 tasks to run.  If dynamic 
allocation has already got all the executors it originally thought it needed 
and they all idle timeout then you have 0 executors, then you can't run the 
tasks anywhere.  Deadlock.  We don't try to reacquire executors until the next 
stage. This patch does address that bug by not allowing you to go to 0 
executors. it keeps enough executors to run all the tasks in parallel.
    the executors idle timeout is based on a time.  If there are other delays 
in the system that cause the scheduler to not schedule tasks fast enough the 
executors will idle timeout.  This could be delays in network, could be because 
driver was Gcing, etc.   if the scheduler doesn't put a task on that executor 
fast enough we can idle timeout it and never get a replacement back.
    The problem with locality is that you don't know unless you give the 
scheduler time to do its logic.  it starts by scheduling things node local and 
eventually falls back to rack and then any.  If you don't allow the executors 
to stay long enough for it to fall back then it can't schedule them there and 
by the nature things will go slower because it can't run the tasks in parallel. 
    NO this is not a problem the user should be solving by increasing the idle 
timeout.  You want things to timeout fairly quickly when there are no tasks 
that could be run on that. Changing the timeout will affect that badly.  and NO 
I don't necessarily want to decrease the locality wait as it could again affect 
job performance.  This again is going to be very job dependent as well as what 
executors I get in this particular instance of the application.  A user should 
not have to work around this issue by changing the configurations. The fact is 
we give back executors and never get more when the spark job could be using 
    It matters if its removed because it would be used shortly and we never get 
another executor back and thus I can never run all my tasks in parallel.  
    This does not ignore the minimum, the minimum is used when there aren't 
enough tasks to actually use all the executors. 

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to