Github user GraceH commented on the pull request:
https://github.com/apache/spark/pull/7888#issuecomment-154322010
Thanks @andrewor14.
Hi @vanzin, Let me give a quick brief to you about the patch and its goal.
There is a bug in dynamic allocation. Since some of the busy executors
might be killed by "mistake", when we met such kind of situation in real-world
deployment frequently.
1. The executor is being idled for 60 seconds, and just marked as to be
killed by dynamic allocation criteria.
2. The scheduler is assigning one/several tasks to that executor. The
listener event not reached that time. (since the listener event only happens
after new tasks assigned synchronously)
3. The executor is killed as planned. But actually, that executor is just
assigned with some tasks. That causes one busy executor is killed by
âmistake".
To solve this problem, one thing is to make that task assignment and
notification synchronized. But this approach is not suitable for current
implementation (listener mechanism).
Here I proposed another way. To add the force control in killExecutor().
For dynamic allocation, we need to check if the executor is busy or not before
really taking the kill action. By doing so, even the listen event not arrives
in time, we can actively rescue certain busy executors (to be killed but with
new tasks assigned). Thru dynamic allocation we should not kill those busy
executors (disable force control).
And meanwhile, we open that force control to the end user (sparkcontext
public API). The end user can decide whether to force kill certain executors .
Please let me understand your thoughts. Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
