joseluisll commented on PR #8903: URL: https://github.com/apache/hadoop/pull/8903#issuecomment-6625569434
> @joseluisll I mean overriding, not excluding. > > > Maven picks the “nearest definition”. > > https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html > > So declaring the slf4j version at the top level in the `dependencyManagement` block overrides the transitive pulled version Hi!. I only believe in the code I execute and see the results in the logs, my two proposals work from the point of view of a downstream application, both declaring in the module and the exclusion. And my tests indicate that under maven 3.9.11 and 3.9.16 what you say does not work, only with maven 4. Also, large downstream projects (HBase, Hive, Flink, Tez, Ozone, Phoenix) depend on hadoop-hdfs directly but pin slf4j-api in their own dependencyManagement, so they’re unaffected either way; Spark uses the shaded client. The risk is for smaller consumers that depend on hadoop-hdfs or hadoop-yarn-client (e.g. for MiniDFSCluster) without managing slf4j themselves. | Project | Uses `hadoop-hdfs` directly? | Uses `hadoop-yarn-client`? | Own slf4j-api pin | Hadoop version today | |---|---|---|---|---| | **HBase** | **Yes, normal scope** in hbase-server, hbase-asyncfs, hbase-mapreduce, hbase-balancer. The test-jar is compile scope in hbase-testing-util. | Only in the bring-your-own-Hadoop assembly, provided scope | 1.7.30, with log4j2's slf4j 1.x binding | 3.4.3 | | Hive | Yes: ql, shims, hcatalog, metastore, streaming, plus many test modules | Yes: ql, shims | 1.7.30, with log4j2's slf4j 1.x binding | 3.4.2 | | Flink | Yes: flink-runtime, flink-yarn, flink-hadoop-fs (excludes Hadoop's slf4j-reload4j) | Yes: flink-yarn | **2.0.20**, with log4j2's slf4j 2 binding | — | | Tez | Mostly test scope; managed in the root POM with `org.eclipse.jetty:*` excluded | Yes, in most modules | 1.7.36 | 3.5.0 | | Ozone | Yes: container-service, cli-debug | No | **2.0.20** (through slf4j-bom) | 3.4.3 | | Phoenix | Test and provided scope | Managed only | 1.7.36 | 3.4.2 | | Spark | **No.** It uses the shaded `hadoop-client-api` and `hadoop-client-runtime` | No | 2.0.17 | 3.5.0 | | Iceberg | Declared in its Gradle version catalog | No | 2.0.18 | 3.4.3 | So following the best practice for dependency management we should declare the dependency where it is used, while the version stays managed in one place in hadoop-project, consistent with the 22 other Hadoop modules that already declare slf4j-api. Exclusions are meant for removing unwanted dependencies rather than choosing versions, and the clean long-term fix is moving Hadoop to slf4j 2. Shall I create another JIRA for that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
