Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/7028#issuecomment-115422594
I think the most important thing is to include the user stack trace
somewhere. Users don't really care what's going on as long as they can identify
a line of code from their codebase in the stack. That said, I see your point
about the current approach being overly magical.
Note that producing an Exception that matches your example would require
wrapping/replacing whichever actual exception with a SparkException (otherwise
we cannot rewrite the message). However, we do have two disjoint options:
1. Keep the munging logic in DAGScheduler.failJobAndIndependentStages and
simply put the "Job Submission" stack trace above the driver stack trace,
included in the Exception's message. This puts the exception a bit "out of
order" but would avoid having to parse the longForm CallSite. We can do this
because this method only throws SparkExceptions anyway.
2. Move the munging logic to DAGScheduler.runJob and keep it as a fake part
of the stack trace. This would allow us to inject the user stack trace into
*all* exceptions thrown as part of the DAGScheduler EventLoop rather than just
ones that induce normal stage fails.
With number 2, we could additionally replace the use of callSite parsing
with the current thread's stack trace, which would make it look a lot less
magical (it's just bridging the two real stack traces across the event loop).
I'd personally prefer 2 for its improved scope, but I don't know enough
about how the call site may differ from the physical stack trace to say if we
should also just use the local stack.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
