[jira] [Comment Edited] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-04 Thread Siyao Meng (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944909#comment-16944909
 ] 

Siyao Meng edited comment on HADOOP-16152 at 10/4/19 11:39 PM:
---

Uploaded patch rev 003:
1. addressing ALL known deprecation issues brought from jetty 9.3->9.4.

TODO:
1. Find out what the shading error is in jenkins and fix it.


was (Author: smeng):
Uploaded patch rev 003:
1. addressing ALL known deprecation issues brought from jetty 9.3->9.4.
2. *TestHttpServer#testPortRanges* failed after jetty 9.4 upgrade, it was 
complaining port 49000 is in use and threw IOException. The suspected root 
cause is that it was throwing BindException before in jetty 9.3, but now it is 
throwing IOException. Solution is to catch IOException as well and wrapping it 
into a BindException in *HttpServer2#bindForPortRange*. It works but not 100% 
it is the correct way.
3. Similar reason of failure for *TestHttpServer#testBindAddress* after jetty 
9.4 upgrade, fixed in *HttpServer2#bindForPortRange*

TODO:
1. Find out what the shading error is in jenkins and fix it.

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, 
> HADOOP-16152.003.patch, HADOOP-16152.004.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-02 Thread Siyao Meng (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943206#comment-16943206
 ] 

Siyao Meng edited comment on HADOOP-16152 at 10/2/19 10:25 PM:
---

The DataNode jetty server max thread issue mentioned 
[above|https://issues.apache.org/jira/browse/HADOOP-16152?focusedCommentId=16942499&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16942499]
 can be solved by adding 1 to *DatanodeHttpServer#HTTP_MAX_THREADS*:
{code}
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
index 86672b403c9..9819fafe291 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
@@ -83,8 +83,9 @@
   // set them to the minimum possible
   private static final int HTTP_SELECTOR_THREADS = 1;
   private static final int HTTP_ACCEPTOR_THREADS = 1;
+  // jetty 9.4: add one extra max thread
   private static final int HTTP_MAX_THREADS =
-  HTTP_SELECTOR_THREADS + HTTP_ACCEPTOR_THREADS + 1;
+  HTTP_SELECTOR_THREADS + HTTP_ACCEPTOR_THREADS + 1 + 1;
   private final HttpServer2 infoServer;
   private final EventLoopGroup bossGroup;
   private final EventLoopGroup workerGroup;
{code}

DataNode works after the change with jetty 9.4 on my Mac. Will post a 003 patch 
after the previous jenkins run finishes.


was (Author: smeng):
The DataNode jetty server max thread issue can be solved by adding 1 to 
DatanodeHttpServer#HTTP_MAX_THREADS:
{code}
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
index 86672b403c9..9819fafe291 100644
--- 
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
+++ 
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
@@ -83,8 +83,9 @@
   // set them to the minimum possible
   private static final int HTTP_SELECTOR_THREADS = 1;
   private static final int HTTP_ACCEPTOR_THREADS = 1;
+  // jetty 9.4: add one extra max thread
   private static final int HTTP_MAX_THREADS =
-  HTTP_SELECTOR_THREADS + HTTP_ACCEPTOR_THREADS + 1;
+  HTTP_SELECTOR_THREADS + HTTP_ACCEPTOR_THREADS + 1 + 1;
   private final HttpServer2 infoServer;
   private final EventLoopGroup bossGroup;
   private final EventLoopGroup workerGroup;
{code}

DataNode works after the change with jetty 9.4 on my Mac. Will post a 003 patch 
after the previous jenkins run finishes.

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch, 
> HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-02 Thread Siyao Meng (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942499#comment-16942499
 ] 

Siyao Meng edited comment on HADOOP-16152 at 10/2/19 9:07 PM:
--

[~weichiu] -I applied the rev 002 patch locally on latest trunk. The compile 
(mvn install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true) also succeeded 
for me. No such "org.eclipse.jetty.server" deprecation warnings. BUT the 
NameNode / DataNode will fail to start, possibly due to the incorrect shading. 
Hmm.-
{code:title=NameNode failed to start w/ patch rev 002}
...
2019-10-01 22:16:10,492 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1185)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:917)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:728)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:985)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:958)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1727)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1792)
Caused by: java.lang.NoSuchMethodError: 
org.eclipse.jetty.server.ResourceContentFactory.(Lorg/eclipse/jetty/util/resource/ResourceFactory;Lorg/eclipse/jetty/http/MimeTypes;Z)V
at 
org.eclipse.jetty.servlet.DefaultServlet.init(DefaultServlet.java:293)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
...
{code}

It turns out I forgot to include "clean" command in maven so the resulting 
distro includes two versions of jetty. And NN possibly picks up the wrong 
version. With command *mvn clean install -Pdist -DskipTests -e 
-Dmaven.javadoc.skip=true*, the NN will start normally now. But the DN would 
fail due to some incorrect jetty configs, guess I need to tune that somehow:
{code}
2019-10-02 14:01:58,227 INFO thread.ThreadPoolBudget: 
SelectorManager@ServerConnector@68034211{HTTP/1.1,[http/1.1]}{localhost:50619} 
requires 1 threads from 
QueuedThreadPool[qtp1396431506]@533bda92{STARTED,3<=3<=3,i=3,r=1,q=0}[ReservedThreadExecutor@1fa1cab1{s=0/1,p=0}]
2019-10-02 14:01:58,229 INFO datanode.DataNode: Waiting up to 30 seconds for 
transfer threads to complete
2019-10-02 14:01:58,229 INFO datanode.DataNode: Gracefully shutting down 
executor service. Waiting max 15 SECONDS
2019-10-02 14:01:58,229 INFO datanode.DataNode: Succesfully shutdown executor 
service
2019-10-02 14:01:58,230 INFO datanode.DataNode: Shutdown complete.
2019-10-02 14:01:58,231 ERROR datanode.DataNode: Exception in secureMain
java.io.IOException: Problem starting http server
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1194)
at 
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:141)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:978)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1438)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:513)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2843)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2749)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2793)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2937)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2961)
Caused by: java.lang.IllegalStateException: Insufficient configured threads: 
required=3 < max=3 for 
QueuedThreadPool[qtp1396431506]@533bda92{STARTED,3<=3<=3,i=3,r=1,q=0}[ReservedThreadExecutor@1fa1cab1{s=0/1,p=0}]
at 
org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:156)
at 
org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130)
at 
org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:182)
at 
org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:255)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at 
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:283)
at 
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)

[jira] [Comment Edited] (HADOOP-16152) Upgrade Eclipse Jetty version to 9.4.x

2019-10-01 Thread Siyao Meng (Jira) 


[ 
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942499#comment-16942499
 ] 

Siyao Meng edited comment on HADOOP-16152 at 10/2/19 5:18 AM:
--

[~jojochuang] I applied the rev 002 patch locally on latest trunk. The compile 
(mvn install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true) also succeeded 
for me. No such "org.eclipse.jetty.server" deprecation warnings. BUT the 
NameNode / DataNode will fail to start, possibly due to the incorrect shading. 
Hmm.
{code:title=NameNode failed to start w/ patch rev 002}
...
2019-10-01 22:16:10,492 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1185)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:917)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:728)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:985)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:958)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1727)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1792)
Caused by: java.lang.NoSuchMethodError: 
org.eclipse.jetty.server.ResourceContentFactory.(Lorg/eclipse/jetty/util/resource/ResourceFactory;Lorg/eclipse/jetty/http/MimeTypes;Z)V
at 
org.eclipse.jetty.servlet.DefaultServlet.init(DefaultServlet.java:293)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
...
{code}


was (Author: smeng):
[~jojochuang] I applied the rev 002 patch locally on latest trunk. The compile 
(mvn install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true) also succeeded 
for me. No such "org.eclipse.jetty.server" deprecation warnings. Hmm.

> Upgrade Eclipse Jetty version to 9.4.x
> --
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
>  Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Yuming Wang
>Assignee: Yuming Wang
>Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some 
> compatibility issues.
> Spark: 
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite: 
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: https://issues.apache.org/jira/browse/HIVE-21211



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org