Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11512#issuecomment-194475748
So if they catch exceptions themselves we have to assume succeeded which is
what we do now. I just hate to add more logic and not really solve the
underlying problem. But unfortunately its really hard for us to know since its
any arbitrary user code. We can't just ask the scheduler if its done.
So I'm thinking about this some more and I'm wondering if we just don't
change the behavior for 2.x to report failed by default. The reason we do it
this way is the SPARK code itself and examples used to System.exit(). We have
removed those quite a few releases back now. People shouldn't be calling
System.exit and should just be calling sc.stop(). 2.x gives us an opportunity
to change that behavior.
@vanzin @pwendell Do you guys have any concerns about this?
if we want to pull this into 1.6.x line I would be ok adding in the signal
handler to fix this one issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
