Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7014#issuecomment-121648671
I played with this locally a bit, and I think that actually defaulting
`Option[Throwable] = None` is covering up some cases where there really is an
exception you should include. I think you should just remove that defaulting
everywhere. It'll lead to a handful of compile errors, but mostly they will be
easy to fix -- either there will be an obvious exception to pass or there won't
...
... except for the default on `ExceptionFailure`. That will lead to a
failure in
[`JsonProtocol.taskEndReasonFromJson`](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala#L764).
Is it possible to turn the actual exception into json in
`JsonProtocol.taskEndReasonToJson`, so that it can be read back in? I was
wondering if getting the full exception back might be useful to anything which
wants to analyze the event logs. I guess its impossible for some general
purpose job to know about whatever user exceptions there are, so maybe there
isn't any good reason for getting the full exception in json anyway.
So I'm leaning towards just putting in `None` for the exception in
`JsonProtocol.taskEndReasonFromJson`, but just wanted to share my thoughts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
