[
https://issues.apache.org/jira/browse/HADOOP-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555999#comment-17555999
]
Viraj Jasani edited comment on HADOOP-18033 at 6/18/22 10:00 PM:
-----------------------------------------------------------------
Thanks [~ayushtkn], yes we should hopefully get the full build QA results in
~24 hrs.
On the shading side, I meant:
{quote}I don't think it was as simple as "completely removing jsr311-api from
Hadoop" would allow us to exclude shading javax.ws.rs-api from both
hadoop-client-minicluster and hadoop-client-runtime
{quote}
I tried this again and the build fails with the same error that I faced earlier:
{code:java}
[INFO] -------< org.apache.hadoop:hadoop-client-check-test-invariants >--------
[INFO] Building Apache Hadoop Client Packaging Invariants for Test
3.4.0-SNAPSHOT [105/112]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @
hadoop-client-check-test-invariants ---
[INFO] Deleting
/Users/vjasani/Documents/src/hadoop-trunk/hadoop/hadoop-client-modules/hadoop-client-check-test-invariants/target
[INFO] Deleting
/Users/vjasani/Documents/src/hadoop-trunk/hadoop/hadoop-client-modules/hadoop-client-check-test-invariants
(includes = [dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @
hadoop-client-check-test-invariants ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/Users/vjasani/Documents/src/hadoop-trunk/hadoop/hadoop-client-modules/hadoop-client-check-test-invariants/target/test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (enforce-banned-dependencies) @
hadoop-client-check-test-invariants ---
[INFO] Adding ignore: module-info
[INFO] Adding ignore: META-INF/versions/*/module-info
[INFO] Adding ignorable dependency: org.apache.hadoop:hadoop-annotations:null
[INFO] Adding ignore: *
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed
with message:
Duplicate classes found:
Found in:
org.apache.hadoop:hadoop-client-minicluster:jar:3.4.0-SNAPSHOT:compile
org.apache.hadoop:hadoop-client-runtime:jar:3.4.0-SNAPSHOT:compile
Duplicate classes:
org/apache/hadoop/shaded/javax/ws/rs/POST.class
org/apache/hadoop/shaded/javax/ws/rs/core/Link$JaxbLink.class
org/apache/hadoop/shaded/javax/ws/rs/NotFoundException.class
org/apache/hadoop/shaded/javax/ws/rs/container/PreMatching.class
org/apache/hadoop/shaded/javax/ws/rs/container/ContainerRequestContext.class
org/apache/hadoop/shaded/javax/ws/rs/core/FeatureContext.class
org/apache/hadoop/shaded/javax/ws/rs/core/HttpHeaders.class
org/apache/hadoop/shaded/javax/ws/rs/PATCH.class
org/apache/hadoop/shaded/javax/ws/rs/sse/OutboundSseEvent$Builder.class
org/apache/hadoop/shaded/javax/ws/rs/core/GenericType.class
org/apache/hadoop/shaded/javax/ws/rs/sse/SseBroadcaster.class
org/apache/hadoop/shaded/javax/ws/rs/core/MediaType$2.class
org/apache/hadoop/shaded/javax/ws/rs/core/StreamingOutput.class
...
...
... {code}
Hence, with the above PR, I have removed exclusion only from
hadoop-client-runtime shade. Now we can confirm that these classes are present
in hadoop-client-runtime but not on hadoop-client-minicluster jar:
{code:java}
$ jar tf
hadoop-client-modules/hadoop-client-runtime/target/hadoop-client-runtime-3.4.0-SNAPSHOT.jar
| grep "AbstractMultivaluedMap"
org/apache/hadoop/shaded/javax/ws/rs/core/AbstractMultivaluedMap.class
$ jar tf
hadoop-client-modules/hadoop-client-minicluster/target/hadoop-client-minicluster-3.4.0-SNAPSHOT.jar
| grep "AbstractMultivaluedMap" {code}
One question for you: how do we determine which client module to shade the new
dependency in? Is it always hadoop-client-runtime (for downstreamers)?
was (Author: vjasani):
Thanks [~ayushtkn], yes we should hopefully get the full build QA results in
~24 hrs.
On the shading side, I meant:
{quote}I don't think it was as simple as "completely removing jsr311-api from
Hadoop" would allow us to exclude shading javax.ws.rs-api from both
hadoop-client-minicluster and hadoop-client-runtime
{quote}
I tried this again and the build fails with the same error that I faced earlier:
{code:java}
[INFO] -------< org.apache.hadoop:hadoop-client-check-test-invariants >--------
[INFO] Building Apache Hadoop Client Packaging Invariants for Test
3.4.0-SNAPSHOT [105/112]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @
hadoop-client-check-test-invariants ---
[INFO] Deleting
/Users/vjasani/Documents/src/hadoop-trunk/hadoop/hadoop-client-modules/hadoop-client-check-test-invariants/target
[INFO] Deleting
/Users/vjasani/Documents/src/hadoop-trunk/hadoop/hadoop-client-modules/hadoop-client-check-test-invariants
(includes = [dependency-reduced-pom.xml], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @
hadoop-client-check-test-invariants ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/Users/vjasani/Documents/src/hadoop-trunk/hadoop/hadoop-client-modules/hadoop-client-check-test-invariants/target/test-dir
[INFO] Executed tasks
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (enforce-banned-dependencies) @
hadoop-client-check-test-invariants ---
[INFO] Adding ignore: module-info
[INFO] Adding ignore: META-INF/versions/*/module-info
[INFO] Adding ignorable dependency: org.apache.hadoop:hadoop-annotations:null
[INFO] Adding ignore: *
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed
with message:
Duplicate classes found:
Found in:
org.apache.hadoop:hadoop-client-minicluster:jar:3.4.0-SNAPSHOT:compile
org.apache.hadoop:hadoop-client-runtime:jar:3.4.0-SNAPSHOT:compile
Duplicate classes:
org/apache/hadoop/shaded/javax/ws/rs/POST.class
org/apache/hadoop/shaded/javax/ws/rs/core/Link$JaxbLink.class
org/apache/hadoop/shaded/javax/ws/rs/NotFoundException.class
org/apache/hadoop/shaded/javax/ws/rs/container/PreMatching.class
org/apache/hadoop/shaded/javax/ws/rs/container/ContainerRequestContext.class
org/apache/hadoop/shaded/javax/ws/rs/core/FeatureContext.class
org/apache/hadoop/shaded/javax/ws/rs/core/HttpHeaders.class
org/apache/hadoop/shaded/javax/ws/rs/PATCH.class
org/apache/hadoop/shaded/javax/ws/rs/sse/OutboundSseEvent$Builder.class
org/apache/hadoop/shaded/javax/ws/rs/core/GenericType.class
org/apache/hadoop/shaded/javax/ws/rs/sse/SseBroadcaster.class
org/apache/hadoop/shaded/javax/ws/rs/core/MediaType$2.class
org/apache/hadoop/shaded/javax/ws/rs/core/StreamingOutput.class
...
...
... {code}
Hence, with the above PR, I have removed exclusion only from
hadoop-client-runtime shade. Now we can confirm that these classes are present
in hadoop-client-runtime but not on hadoop-client-minicluster jar:
{code:java}
$ jar tf
hadoop-client-modules/hadoop-client-runtime/target/hadoop-client-runtime-3.4.0-SNAPSHOT.jar
| grep "AbstractMultivaluedMap"
org/apache/hadoop/shaded/javax/ws/rs/core/AbstractMultivaluedMap.class
$ jar tf
hadoop-client-modules/hadoop-client-minicluster/target/hadoop-client-minicluster-3.4.0-SNAPSHOT.jar
| grep "AbstractMultivaluedMap" {code}
> Upgrade fasterxml Jackson to 2.13.0
> -----------------------------------
>
> Key: HADOOP-18033
> URL: https://issues.apache.org/jira/browse/HADOOP-18033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Reporter: Akira Ajisaka
> Assignee: Viraj Jasani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Spark 3.2.0 depends on Jackson 2.12.3. Let's upgrade to 2.12.5 (2.12.x latest
> as of now) or upper.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
