[
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16413974#comment-16413974
]
Sahil Takiar commented on HIVE-18831:
-------------------------------------
Hey [~lirui]. I think the benefits of the re-factoring are as follows:
* Doesn't require the manual parsing from HIVE-15237, we can just use
{{Throwables#getRootCause}}, which feels a bit safer
* It makes the code easier to understand; we aren't really sending an
{{Exception}} from the {{RemoteDriver}} to the {{SparkClient}}, we are actually
just sending a string representation of the stack-trace, so trying to wrap it
inside an {{Exception}} makes the code confusing; Spark itself takes a similar
approach (see SPARK-8625 and Spark's {{ExceptionFailure}} which is where I got
the idea from)
* Differentiating between errors from the job monitor vs. the Spark job is
useful because we way want to treat them differently; for example, all the OOM
parsing logic should be specific to exceptions from the job itself; we may also
want to prefer to propagate exceptions from the Spark job vs. the job monitor,
as the Spark job exceptions are probably more relevant
Beyond that, the patch itself has a few other benefits:
* Unifying the exception message for task failures
* Propagating the root cause of a job failure, rather than telling users to
look through the logs
* Lots of tests
> Differentiate errors that are thrown by Spark tasks
> ---------------------------------------------------
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch,
> HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch, HIVE-18831.7.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we
> don't differentiate between errors from HS2 / RSC vs. errors thrown by
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its
> difficult to know what part of the excution threw the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
