GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/12433
[SPARK-14676] Wrap and re-throw Await.result exceptions in order to capture
full stacktrace
When `Await.result` throws an exception which originated from a different
thread, the resulting stacktrace doesn't include the path leading to the
`Await.result` call itself, making it difficult to identify the impact of these
exceptions. For example, I've seen cases where broadcast cleaning errors
propagate to the main thread and crash it but the resulting stacktrace doesn't
include any of the main thread's code, making it difficult to pinpoint which
exception crashed that thread.
This patch addresses this issue by explicitly catching, wrapping, and
re-throwing exceptions that are thrown by `Await.result`.
I tested this manually using
https://github.com/JoshRosen/spark/commit/16b31c825197ee31a50214c6ba3c1df08148f403,
a patch which reproduces an issue where an RPC exception which occurs while
unpersisting RDDs manages to crash the main thread without any useful
stacktrace, and verified that informative, full stacktraces were generated.
/cc @rxin @nongli @yhuai @anabranch
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark
wrap-and-rethrow-await-exceptions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12433.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12433
----
commit 23e7467de7ba39b16dc920031111c842ac1156f8
Author: Josh Rosen <[email protected]>
Date: 2016-04-15T22:48:06Z
Wrap exceptions thrown by Await.result().
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
