voonhous commented on PR #18177:
URL: https://github.com/apache/hudi/pull/18177#issuecomment-4956090323
Follow-up: the Hive exclusion was only half of it. CI failed again with the
**same** `NoSuchMethodError`, this time on `SPARK_PROFILE: spark3.3`.
### Second source of the collision
Spark only switched to `log4j-slf4j2-impl` in **3.4**.
`spark-core_2.12:3.3.x` still depends on `log4j-slf4j-impl`, so under the
`spark3.3` profile it re-introduces the duplicate
`org.apache.logging.slf4j.Log4jLoggerFactory` and fails identically:
```
NoSuchMethodError: Log4jLoggerFactory.<init>(Log4jMarkerFactory)
at org.apache.logging.slf4j.SLF4JServiceProvider.initialize
```
So there are exactly two sources of the slf4j 1.x binding, both now excluded:
| source | affects |
| --- | --- |
| `hive-shims-common` (Hive 2.3.10) | all profiles |
| `spark-core` (Spark 3.3 only) | `spark3.3` |
### Fix (latest commit)
Exclude `log4j-slf4j-impl` from both `spark-core` entries in
`dependencyManagement` (main + `tests` classifier), next to the `org.slf4j:*`
and `log4j:log4j` exclusions already there. Spark 3.4+ is unaffected -- it
ships the slf4j2 binding already.
### Verified
- Zero `log4j-slf4j-impl` on the dependency tree for **all** profiles:
`spark3.3`, `spark3.4`, `spark3.5`, `spark4.0`, `spark4.1`, `spark4.2`,
`flink1.18`, `flink1.19`, `flink1.20`, `flink2.0`, `flink2.1`, default.
- `hudi-examples-spark` tests (`TestHoodieSparkQuickstart`,
`TestRandomJsonSource`) pass locally on both `spark3.3` and `spark3.5`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
