[
https://issues.apache.org/jira/browse/SPARK-51660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-51660:
-------------------------------------------
Assignee: Rob Reeves
> Spark can only run one task per executor when MDC can't load
> ------------------------------------------------------------
>
> Key: SPARK-51660
> URL: https://issues.apache.org/jira/browse/SPARK-51660
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.1.0
> Reporter: Rob Reeves
> Assignee: Rob Reeves
> Priority: Major
> Labels: pull-request-available
>
> In SPARK-37843, logic was added to gracefully fallback to not logging with
> MDC when MDC cannot be initialized. The change there does not fully fix the
> issue. It catches NoSuchFieldError, which is thrown during MDC
> initialization. This will work for the first task. If a second task runs it
> will not throw a NoSuchFieldError. Instead it throws NoClassDefFoundError.
> When that happens the Spark job will hang forever.
> To fix this, the logic should be enhanced so it gracefully falls back for all
> tasks. We can also improve the noisiness of the logs falling back back once
> and disabling MDC instead of trying for every task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
