[
https://issues.apache.org/jira/browse/SPARK-36919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-36919.
----------------------------------
Fix Version/s: 3.1.3
3.2.0
3.0.4
Resolution: Fixed
Issue resolved by pull request 34167
[https://github.com/apache/spark/pull/34167]
> Make BadRecordException serializable
> ------------------------------------
>
> Key: SPARK-36919
> URL: https://issues.apache.org/jira/browse/SPARK-36919
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.2.0, 3.3.0, 3.2.1
> Reporter: Tianhan Hu
> Assignee: Tianhan Hu
> Priority: Minor
> Fix For: 3.0.4, 3.2.0, 3.1.3
>
>
> Migrating a Spark application from 2.4.x to 3.1.x and finding a difference in
> the exception chaining behavior. In a case of parsing a malformed CSV, where
> the root cause exception should be {{Caused by: java.lang.RuntimeException:
> Malformed CSV record}}, only the top level exception is kept, and all lower
> level exceptions and root cause are lost. Thus, when we call
> {{ExceptionUtils.getRootCause}} on the exception, we still get itself.
> The reason for the difference is that {{RuntimeException}} is wrapped in
> {{BadRecordException}}, which has unserializable fields. When we try to
> serialize the exception from tasks and deserialize from scheduler, the
> exception is lost.
> This PR makes unserializable fields of {{BadRecordException}} transient, so
> the rest of the exception could be serialized and deserialized properly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
