[
https://issues.apache.org/jira/browse/TEZ-4114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033215#comment-17033215
]
László Bodor edited comment on TEZ-4114 at 2/9/20 2:34 PM:
-----------------------------------------------------------
[~ashutoshc], [~jeagles]: could you please take a look at [^TEZ-4114.03.patch]
?
some thoughts about the patch, which could help understand my point:
1. removed compile scoped jetty dependencies (excludes)
because of this you should not see any compile scoped jetty 9.3 deps (brought
from hadoop) by running:
{code}
mvn dependency:tree
{code}
2. removed jetty from root level depencencyManagement, and specified jetty
version in tez-dag, as it caused unit test failures of TestTezCommonUtils, by
messing up test classpath like below (note "version managed from" parts):
{code}
[DEBUG] org.apache.hadoop:hadoop-hdfs:jar:tests:3.0.3:test
[DEBUG] org.eclipse.jetty:jetty-server:jar:9.4.26.v20200117:compile
(version managed from 9.3.19.v20170502 by org.apache.tez:tez:0.10.1-SNAPSHOT)
[DEBUG] org.eclipse.jetty:jetty-http:jar:9.4.26.v20200117:compile
[DEBUG] org.eclipse.jetty:jetty-io:jar:9.4.26.v20200117:compile
[DEBUG] org.eclipse.jetty:jetty-util:jar:9.4.26.v20200117:compile
(version managed from 9.3.19.v20170502 by org.apache.tez:tez:0.10.1-SNAPSHOT)
[DEBUG] org.eclipse.jetty:jetty-util-ajax:jar:9.3.19.v20170502:test
[DEBUG] com.sun.jersey:jersey-server:jar:1.19:test
[DEBUG] commons-daemon:commons-daemon:jar:1.0.13:test
[DEBUG] io.netty:netty:jar:3.10.5.Final:compile
[DEBUG] io.netty:netty-all:jar:4.0.23.Final:test
[DEBUG] org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:test
[DEBUG] org.apache.hadoop:hadoop-common:jar:tests:3.0.3:test
[DEBUG] org.eclipse.jetty:jetty-servlet:jar:9.3.19.v20170502:test
[DEBUG] org.eclipse.jetty:jetty-security:jar:9.3.19.v20170502:test
[DEBUG] org.eclipse.jetty:jetty-webapp:jar:9.3.19.v20170502:test
[DEBUG] org.eclipse.jetty:jetty-xml:jar:9.3.19.v20170502:test
[DEBUG] javax.servlet.jsp:jsp-api:jar:2.1:test
{code}
regarding test classpath, only 9.3 (current hadoop jetty version) should be seen
this log was given by:
{code}
mvn clean install -pl ./tez-api -am -Dtest=TestTezCommonUtils -X | tee test.log
{code}
3. removed mortbay jetty exclusion as they didn't have an effect, see the same
for hive: HIVE-21048
can be checked by:
{code}
grep -iRH "org.mortbay.jetty" --include="pom.xml"
{code}
4. shaded jetty in tez-dag
in dependency tree you can see that the only submodule which depends directly
on jetty is tez-dag, so shading should be taken care of only there
shaded jar is 3.0M compared to original 1.4M
{code}
-rw-r--r-- 1 abstractdog abstractdog 1.4M Feb 9 15:13
original-tez-dag-0.10.1-SNAPSHOT.jar
...
-rw-r--r-- 1 abstractdog abstractdog 3.0M Feb 9 15:13
tez-dag-0.10.1-SNAPSHOT.jar
{code}
was (Author: abstractdog):
[~ashutoshc], [~jeagles]: could you please take a look at [^TEZ-4114.03.patch]
?
some thoughts about the patch, which could help understand my point:
1. removed compile scoped jetty dependencies (excludes)
because of this you should not see any compile scoped jetty 9.3 deps (brought
from hadoop) by running:
{code}
mvn dependency:tree
{code}
2. removed jetty from root level depencencyManagement, and specified jetty
version in tez-dag, as it caused unit test failures of TestTezCommonUtils, by
messing up test classpath like below (note "version managed from" parts):
{code}
[DEBUG] org.apache.hadoop:hadoop-hdfs:jar:tests:3.0.3:test
[DEBUG] org.eclipse.jetty:jetty-server:jar:9.4.26.v20200117:compile
(version managed from 9.3.19.v20170502 by org.apache.tez:tez:0.10.1-SNAPSHOT)
[DEBUG] org.eclipse.jetty:jetty-http:jar:9.4.26.v20200117:compile
[DEBUG] org.eclipse.jetty:jetty-io:jar:9.4.26.v20200117:compile
[DEBUG] org.eclipse.jetty:jetty-util:jar:9.4.26.v20200117:compile
(version managed from 9.3.19.v20170502 by org.apache.tez:tez:0.10.1-SNAPSHOT)
[DEBUG] org.eclipse.jetty:jetty-util-ajax:jar:9.3.19.v20170502:test
[DEBUG] com.sun.jersey:jersey-server:jar:1.19:test
[DEBUG] commons-daemon:commons-daemon:jar:1.0.13:test
[DEBUG] io.netty:netty:jar:3.10.5.Final:compile
[DEBUG] io.netty:netty-all:jar:4.0.23.Final:test
[DEBUG] org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:test
[DEBUG] org.apache.hadoop:hadoop-common:jar:tests:3.0.3:test
[DEBUG] org.eclipse.jetty:jetty-servlet:jar:9.3.19.v20170502:test
[DEBUG] org.eclipse.jetty:jetty-security:jar:9.3.19.v20170502:test
[DEBUG] org.eclipse.jetty:jetty-webapp:jar:9.3.19.v20170502:test
[DEBUG] org.eclipse.jetty:jetty-xml:jar:9.3.19.v20170502:test
[DEBUG] javax.servlet.jsp:jsp-api:jar:2.1:test
{code}
regarding test classpath, only 9.3 (current hadoop jetty version) should be seen
this log was given by:
{code}
mvn clean install -pl ./tez-api -am -Dtest=TestTezCommonUtils -X | tee test.log
{code}
3. removed mortbay jetty exclusion as they didn't have an effect, see the same
for hive: HIVE-21048
4. shaded jetty in tez-dag
in dependency tree you can see that the only submodule which depends directly
on jetty is tez-dag, so shading should be taken care of only there
shaded jar is 3.0M compared to original 1.4M
{code}
-rw-r--r-- 1 abstractdog abstractdog 1.4M Feb 9 15:13
original-tez-dag-0.10.1-SNAPSHOT.jar
...
-rw-r--r-- 1 abstractdog abstractdog 3.0M Feb 9 15:13
tez-dag-0.10.1-SNAPSHOT.jar
{code}
> Upgrade to Jetty 9.4
> --------------------
>
> Key: TEZ-4114
> URL: https://issues.apache.org/jira/browse/TEZ-4114
> Project: Apache Tez
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4114.01.patch, TEZ-4114.02.patch, TEZ-4114.03.patch
>
>
> Hadoop already did it in: https://issues.apache.org/jira/browse/HADOOP-16152
> Hive: "in progress", only abandoned jiras
> https://issues.apache.org/jira/browse/HIVE-21961
> https://issues.apache.org/jira/browse/HIVE-21211
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
