Eric Payne created HDFS-7224:
--------------------------------
Summary: Allow reuse of NN connections via webhdfs
Key: HDFS-7224
URL: https://issues.apache.org/jira/browse/HDFS-7224
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.0
Reporter: Eric Payne
Assignee: Eric Payne
In very large clusters, the webhdfs client could get bind exceptions because it
runs out of ephemeral
ports. This could happen when using webhdfs to talk to the NN in order to do
list globbing of a
huge amount of files.
WebHdfsFileSystem#jsonParse gets the input/error stream from the connection,
but never closes the stream. Since it's not closed, the JVM thinks the stream
may still
be transferring data, so the next time through this code, it has to get a new
connection
rather than reusing an existing one.
The lack of connection reuse has poor latency and adds too much overhead to the
NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
