[
https://issues.apache.org/jira/browse/HADOOP-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942499#comment-16942499
]
Siyao Meng edited comment on HADOOP-16152 at 10/2/19 9:07 PM:
--------------------------------------------------------------
[~weichiu] -I applied the rev 002 patch locally on latest trunk. The compile
(mvn install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true) also succeeded
for me. No such "org.eclipse.jetty.server" deprecation warnings. BUT the
NameNode / DataNode will fail to start, possibly due to the incorrect shading.
Hmm.-
{code:title=NameNode failed to start w/ patch rev 002}
...
2019-10-01 22:16:10,492 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1185)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:917)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:728)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:958)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1727)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1792)
Caused by: java.lang.NoSuchMethodError:
org.eclipse.jetty.server.ResourceContentFactory.<init>(Lorg/eclipse/jetty/util/resource/ResourceFactory;Lorg/eclipse/jetty/http/MimeTypes;Z)V
at
org.eclipse.jetty.servlet.DefaultServlet.init(DefaultServlet.java:293)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
...
{code}
It turns out I forgot to include "clean" command in maven so the resulting
distro includes two versions of jetty. And NN possibly picks up the wrong
version. With command *mvn clean install -Pdist -DskipTests -e
-Dmaven.javadoc.skip=true*, the NN will start normally now. But the DN would
fail due to some incorrect jetty configs, guess I need to tune that somehow:
{code}
2019-10-02 14:01:58,227 INFO thread.ThreadPoolBudget:
SelectorManager@ServerConnector@68034211{HTTP/1.1,[http/1.1]}{localhost:50619}
requires 1 threads from
QueuedThreadPool[qtp1396431506]@533bda92{STARTED,3<=3<=3,i=3,r=1,q=0}[ReservedThreadExecutor@1fa1cab1{s=0/1,p=0}]
2019-10-02 14:01:58,229 INFO datanode.DataNode: Waiting up to 30 seconds for
transfer threads to complete
2019-10-02 14:01:58,229 INFO datanode.DataNode: Gracefully shutting down
executor service. Waiting max 15 SECONDS
2019-10-02 14:01:58,229 INFO datanode.DataNode: Succesfully shutdown executor
service
2019-10-02 14:01:58,230 INFO datanode.DataNode: Shutdown complete.
2019-10-02 14:01:58,231 ERROR datanode.DataNode: Exception in secureMain
java.io.IOException: Problem starting http server
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1194)
at
org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.<init>(DatanodeHttpServer.java:141)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:978)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1438)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:513)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2843)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2749)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2793)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2937)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2961)
Caused by: java.lang.IllegalStateException: Insufficient configured threads:
required=3 < max=3 for
QueuedThreadPool[qtp1396431506]@533bda92{STARTED,3<=3<=3,i=3,r=1,q=0}[ReservedThreadExecutor@1fa1cab1{s=0/1,p=0}]
at
org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:156)
at
org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130)
at
org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:182)
at
org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:255)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at
org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:283)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.server.Server.doStart(Server.java:385)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1158)
... 9 more
2019-10-02 14:01:58,236 INFO util.ExitUtil: Exiting with status 1:
java.io.IOException: Problem starting http server
2019-10-02 14:01:58,238 INFO datanode.DataNode: SHUTDOWN_MSG:
{code}
was (Author: smeng):
[~weichiu] I applied the rev 002 patch locally on latest trunk. The compile
(mvn install -Pdist -DskipTests -e -Dmaven.javadoc.skip=true) also succeeded
for me. No such "org.eclipse.jetty.server" deprecation warnings. BUT the
NameNode / DataNode will fail to start, possibly due to the incorrect shading.
Hmm.
{code:title=NameNode failed to start w/ patch rev 002}
...
2019-10-01 22:16:10,492 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Unable to initialize WebAppContext
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1185)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:170)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:917)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:728)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:958)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1727)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1792)
Caused by: java.lang.NoSuchMethodError:
org.eclipse.jetty.server.ResourceContentFactory.<init>(Lorg/eclipse/jetty/util/resource/ResourceFactory;Lorg/eclipse/jetty/http/MimeTypes;Z)V
at
org.eclipse.jetty.servlet.DefaultServlet.init(DefaultServlet.java:293)
at javax.servlet.GenericServlet.init(GenericServlet.java:244)
...
{code}
> Upgrade Eclipse Jetty version to 9.4.x
> --------------------------------------
>
> Key: HADOOP-16152
> URL: https://issues.apache.org/jira/browse/HADOOP-16152
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 3.2.0
> Reporter: Yuming Wang
> Assignee: Siyao Meng
> Priority: Major
> Attachments: HADOOP-16152.002.patch, HADOOP-16152.002.patch,
> HADOOP-16152.v1.patch
>
>
> Some big data projects have been upgraded Jetty to 9.4.x, which causes some
> compatibility issues.
> Spark:
> [https://github.com/apache/spark/blob/02a0cdea13a5eebd27649a60d981de35156ba52c/pom.xml#L146]
> Calcite:
> [https://github.com/apache/calcite/blob/avatica-1.13.0-rc0/pom.xml#L87]
> Hive: HIVE-21211
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
