[
https://issues.apache.org/jira/browse/SPARK-34939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-34939.
-----------------------------------
Fix Version/s: 3.0.3
3.1.2
2.4.8
3.2.0
Resolution: Fixed
Issue resolved by pull request 32033
[https://github.com/apache/spark/pull/32033]
> Throw fetch failure exception when unable to deserialize broadcasted map
> statuses
> ---------------------------------------------------------------------------------
>
> Key: SPARK-34939
> URL: https://issues.apache.org/jira/browse/SPARK-34939
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.7, 3.0.2, 3.2.0, 3.1.1
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Fix For: 3.2.0, 2.4.8, 3.1.2, 3.0.3
>
>
> One customer encountered application error. From the log, it is caused by
> accessing non-existing broadcasted value. The broadcasted value is map
> statuses. There is a race-condition.
> After map statuses are broadcasted and the executors obtain serialized
> broadcasted map statuses. If any fetch failure happens after, Spark scheduler
> invalidates cached map statuses and destroy broadcasted value of the map
> statuses. Then any executor trying to deserialize serialized broadcasted map
> statuses and access broadcasted value, {{IOException}} will be thrown.
> Currently we don't catch it in {{MapOutputTrackerWorker}} and above exception
> will fail the application.
> Normally we should throw a fetch failure exception for such case and let
> Spark scheduler handle this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
