[
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419458#comment-16419458
]
Sahil Takiar commented on HIVE-18831:
-------------------------------------
Hey [~lirui]
Thanks for the suggestion, serializing into a byte array fixed the issues I was
seeing, the unit tests now pass!
For (2), that would make the serialization logic simpler, and we could avoid
{{JobResultSerializer#convertToSerializableSparkException}}, but that would
make the client side a bit messier. Specifically, the logic in {{SparkTask}}
for parsing {[sparkJobStatus.getSparkJobException}}. If we want to enforce that
{{getSparkJobException}} returns a {{Throwable}} then we would have to wrap the
full-stack trace in a {{SparkException}} (which is what we do now). The issue
is that we try to get the message from the root cause and stick in the
{{HiveException}} message. This would cause the full stack-trace to be placed
in the {{HiveException}} message, and the full-stack would be displayed to the
user.
Instead of using {{SparkException}} in {{#convertToSerializableSparkException}}
we can create something new like {{SafeSerializableExceptionWrapper}}.
> Differentiate errors that are thrown by Spark tasks
> ---------------------------------------------------
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch,
> HIVE-18831.3.patch, HIVE-18831.4.patch, HIVE-18831.6.patch,
> HIVE-18831.7.patch, HIVE-18831.8.WIP.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we
> don't differentiate between errors from HS2 / RSC vs. errors thrown by
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its
> difficult to know what part of the excution threw the exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
