[
https://issues.apache.org/jira/browse/HADOOP-18033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555973#comment-17555973
]
Viraj Jasani edited comment on HADOOP-18033 at 6/18/22 8:19 PM:
----------------------------------------------------------------
{quote}The duplicate class exception that we saw here was actually an alarm
here that these two dependencies can't stay in peace together, but we got away
with that by an exclude...
{quote}
I don't think it was as simple as "completely removing jsr311-api from Hadoop"
would allow us to exclude shading javax.ws.rs-api from both
hadoop-client-minicluster and hadoop-client-runtime. I have already tried this
before, it doesn't work AFAIK. At least, one of them would have to keep the
exclusion on.
HADOOP-15983 has upgraded all com.sun.jersey dependencies (jersey-core,
jersey-servlet etc) to the latest version in 1.x line and the latest version of
jersey-core pulls-in jsr311-api with it:
{code:java}
[INFO] | +- com.sun.jersey:jersey-core:jar:1.19.4:compile
[INFO] | | \- javax.ws.rs:jsr311-api:jar:1.1.1:compile {code}
I don't think without exclusion (and maybe some additional code change, if
JAX-RS 1.x and 2.x incompatibilities are in use), we might be able to get rid
of jsr311-api. I would expect the same for Tez and other dependencies as well.
Tez and other dependencies also can explore the similar path of excluding
jsr311-api completely and only rely on JAX-RS 2.x based javax.ws.rs-api
(specifically if already using jersey 2.x release versions).
jsr311-api is the official spec jar for the JAX-RS 1.x line and the latest
central release available is from Nov, 2009 (too old)
[https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api]
whereas javax.ws.rs-api is jar for JAX-RS 2.x line
[https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api] (latest
version from 2018), hence we can expect more upgraded thirdparty libraries
(just like Jackson2) having dependency on javax.ws.rs-api and less on
jsr311-api. So all downstreamers (Tez, Hadoop, Spark) should try to get rid of
jsr311-api anyways, totally agree here. One dependency doesn't necessarily have
to wait for another to remove it, for instance, Tez can go ahead with exclusion
of jsr311-api even before upgrading to Hadoop 3.3.4 because if not Hadoop, some
of it's other dependencies (like jersey-core latest version, as mentioned
above) would likely anyways pull it in transitively.
On the other hand, let Hadoop also get rid of jsr311-api. But I am pretty sure,
removing it won't solve shading issue completely. Will come up with patch
because I do recall I have already tried this as part of this Jira only. We can
also run full build QA (all modules) and I can manually verify HDFS, MapReduce
and ATSv2 working on pseudo-distributed mode.
was (Author: vjasani):
{quote}The duplicate class exception that we saw here was actually an alarm
here that these two dependencies can't stay in peace together, but we got away
with that by an exclude...
{quote}
I don't think it was as simple as "completely removing jsr311-api from Hadoop"
would allow us to exclude shading javax.ws.rs-api from both
hadoop-client-minicluster and hadoop-client-runtime. I have already tried this
before, it doesn't work AFAIK. At least, one of them would have to keep the
exclusion on (just like say for instance, hadoop-annotation I guess).
HADOOP-15983 has upgraded all com.sun.jersey dependencies (jersey-core,
jersey-servlet etc) to the latest version and still the latest version of
jersey-core pulls-in jsr311-api with it:
{code:java}
[INFO] | +- com.sun.jersey:jersey-core:jar:1.19.4:compile
[INFO] | | \- javax.ws.rs:jsr311-api:jar:1.1.1:compile {code}
I don't think without exclusion (and maybe some additional code change, if
JAX-RS 1.x and 2.x incompatibilities are in use), we might be able to get rid
of jsr311-api. I would expect the same for Tez and other dependencies as well.
Tez and other dependencies also can explore the similar path of excluding
jsr311-api completely and only rely on JAX-RS 2.x based javax.ws.rs-api.
jsr311-api is the official spec jar for the JAX-RS 1.x line and the latest
central release available is from Nov, 2009 (too old)
[https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api]
whereas javax.ws.rs-api is jar for JAX-RS 2.x line
[https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api] (latest
version from 2018), hence we can expect more upgraded thirdparty libraries
(just like Jackson2) having dependency on javax.ws.rs-api and less on
jsr311-api. So all downstreamers (Tez, Hadoop, Spark) should try to get rid of
jsr311-api anyways, totally agree here. One dependency doesn't necessarily have
to wait for another to remove it, for instance, Tez can go ahead with exclusion
of jsr311-api even before upgrading to Hadoop 3.3.4 because if not Hadoop, some
of it's other dependencies (like jersey-core latest version, as mentioned
above) would likely anyways pull it in transitively.
On the other hand, let Hadoop also get rid of jsr311-api. But I am pretty sure,
removing it won't solve shading issue completely. Will come up with patch soon
because I do recall I have already tried this as part of this Jira only. We can
also run full build QA (all modules) and I can manually verify HDFS, MapReduce
and ATSv2 working on pseudo-distributed mode.
> Upgrade fasterxml Jackson to 2.13.0
> -----------------------------------
>
> Key: HADOOP-18033
> URL: https://issues.apache.org/jira/browse/HADOOP-18033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Reporter: Akira Ajisaka
> Assignee: Viraj Jasani
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.2
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Spark 3.2.0 depends on Jackson 2.12.3. Let's upgrade to 2.12.5 (2.12.x latest
> as of now) or upper.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
