[
https://issues.apache.org/jira/browse/HDFS-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15051427#comment-15051427
]
Xiao Chen commented on HDFS-7553:
---------------------------------
Not sure if [~xieliang007] is still working on this, but I want to self-promote
patch 2 with a way to reproduce. Attached is the reproduce patch which includes
patch 2, that fails with the bind exception when restarting NN. Uncommenting
the {{httpServer.join();}} at {{NameNode.java#932}} makes the test pass. The
test case I ran was {{testNfsUpgrade}}.
In the repro patch, the exception is thrown from {{HttpServer2.openListeners}},
instead of the inner jetty + net classes. But IMHO this already proves that we
need a {{join}} to have a successful immediate NN restart. (Arguably though,
with this the NN shutdown will be slower, and potentially hang if somehow the
webserver hangs in its shutdown. I think this would be the webserver's bug, and
could be enhanced in a separate JIRA. E.g. add a timeout to the NN shutdown)
Thanks [~wheat9] for helping understand the problem, and [~cnauroth] for the
initial thought - turns out to be exactly the cause! Please feel free try out
the repro patch and review patch 2.
> fix the TestDFSUpgradeWithHA due to BindException
> -------------------------------------------------
>
> Key: HDFS-7553
> URL: https://issues.apache.org/jira/browse/HDFS-7553
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
> Affects Versions: 2.7.0
> Reporter: Liang Xie
> Assignee: Liang Xie
> Labels: BB2015-05-TBR
> Attachments: HDFS-7553-001.txt, HDFS-7553.002.patch,
> HDFS-7553.repro.patch
>
>
> see
> https://builds.apache.org/job/PreCommit-HDFS-Build/9092//testReport/org.apache.hadoop.hdfs.server.namenode.ha/TestDFSUpgradeWithHA/testNfsUpgrade/
> :
> Error Message
> Port in use: localhost:57896
> Stacktrace
> java.net.BindException: Port in use: localhost:57896
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
> at
> org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:868)
> at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:809)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:704)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:591)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:763)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:747)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1443)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1815)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1796)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA.testNfsUpgrade(TestDFSUpgradeWithHA.java:285)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
