Daryn Sharp created HDFS-7607:
---------------------------------
Summary: Use random rack-local node for webhdfs opens to avoid OOM
on DNs
Key: HDFS-7607
URL: https://issues.apache.org/jira/browse/HDFS-7607
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode, webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Webhdfs currently redirects a client to the DN that physically has one of the
replicas. Unlike the hdfs data streamer protocol which can easily handle
hundreds or thousands of connections, jetty has poor performance under heavy
load. Webhdfs clients can easily overwhelm the DNs and likely cause OOMs or
excessive GC.
The NN should redirect the client to a rack-local location to distribute the
webhdfs load across multiple hosts. The rack can then use the lightweight
streamer protocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
