Rob Reeves created SPARK-51660:
----------------------------------
Summary: Spark can only run one task per executor when MDC can't
load
Key: SPARK-51660
URL: https://issues.apache.org/jira/browse/SPARK-51660
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.1.0
Reporter: Rob Reeves
In SPARK-37843, logic was added to gracefully fallback to not logging with MDC
when MDC cannot be initialized. The change there does not fully fix the issue.
It catches NoSuchFieldError, which is thrown during MDC initialization. This
will work for the first task. If a second task runs it will not throw a
NoSuchFieldError. Instead it throws NoClassDefFoundError. When that happens the
Spark job will hang forever.
To fix this, the logic should be enhanced so it gracefully falls back for all
tasks. We can also improve the noisiness of the logs falling back back once and
disabling MDC instead of trying for every task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
