Github user zhuoliu commented on the pull request:
https://github.com/apache/spark/pull/9946#issuecomment-175121155
Sure.
A brief summary is that: (picked a few conclusive points from above):
"At the point we call System.exit here all user code is done and we are
terminating. If there is something you know of that the user should be allowed
to do after this then we should create something like a shutdown hook so the
user can properly clean it up.
A timeout would probably work here but I'm not fond of it here, it is going
to delay it for everyone and you don't give the resources back as quickly. You
could also end up in the same situation where if you choose a timeout, say 3
seconds and if this theoretical user code didn't finish within that 3 seconds
you would kill it anyway."
"I also don't buy the existence of any valid user case for a user thread to
block container exit,"
"I'm saying that it's the user's fault if his application depends on that
non-predictable behavior. We're talking milliseconds here, that might be
affected by everything from YARN's internal code to network delays to kernel
scheduling. There's absolutely no argument for someone depending on that for
the correct behavior of their application."'
"I'm on board with this if it's true that YARN virtually immediately kills
a JVM like this if it's not done by the time the NM thinks it's done. Then
indeed regardless of what it does to an app that's tardy in cleaning up, it's
not a materially different behavior. If it helps handle another bad app
behavior, there's no downside to doing that."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
