Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/7786
If you want to add a new config for the "kill preempted containers"
functionality that would probably be an acceptable compromise.
---
Github user chemikadze commented on the issue:
https://github.com/apache/spark/pull/7786
@vanzin If those would be implemented, would it have any change to get
merged? We use preemption quite a lot and current behavior is not the best we
can get: logs sometimes getting overfilled
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/7786
@vazin, sorry for the delay.
I think your concern is valid that this change will probably harm the
performance since 15 seconds (preemption waiting) is not a short time.
But
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/7786
@jerryshao do you plan to implement either of the above suggestions?
otherwise we should probably close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/7786
> But I'm just trying to point out that the current change doesn't really
make things better. Without killing the executor, you'll still be holding on to
resources, except now you wouldn't be
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/7786
@steveloughran
> I suspect that if you get told you are being pre-empted, you aren't
likely to get containers elsewhere
That's very possible. But I'm just trying to point out that
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/7786
@vanzin I suspect that if you get told you are being pre-empted, you aren't
likely to get containers elsewhere âpre-emption is a sign of demand being too
high, and your queue lower priority.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/7786
> by default Yarn will preempt the container 15 seconds after the warning
That's a long time and you can run a lot of tasks in that time. Unless
Spark actively goes and gets rid of these
