Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/11512#issuecomment-193855548
So I think the problem with this is still that we don't know the real state
of the application. There is still a race condition here where the users
application and sparkcontext could be succeeded (ie wrote output and was about
to finish) but if we get a SIGTERM at the wrong point we mark it as failed
rather then succeeded.
Normally if Spark was actually running anything the DAGScheduler would
throw an exception. I think your use case here is the driver is sleeping and
not running anything distributed so the scheduler doesn't throw, it just calls
sc.stop(). I would rather explore having the scheduler throw if it has any
jobs, including inactive ones or allowing the application master to know the
state of the Spark jobs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
