[
https://issues.apache.org/jira/browse/HDFS-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277110#comment-14277110
]
Daryn Sharp commented on HDFS-7607:
-----------------------------------
Yes, but it will probably take a few weeks. Maybe we should retarget for 2.6.1
in case that release is made? We've been running with this patch since last
fall otherwise webhdfs would cause cluster meltdowns, so others may find it
useful.
> Use random rack-local node for webhdfs opens to avoid OOM on DNs
> ----------------------------------------------------------------
>
> Key: HDFS-7607
> URL: https://issues.apache.org/jira/browse/HDFS-7607
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, webhdfs
> Affects Versions: 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
>
> Webhdfs currently redirects a client to the DN that physically has one of the
> replicas. Unlike the hdfs data streamer protocol which can easily handle
> hundreds or thousands of connections, jetty has poor performance under heavy
> load. Webhdfs clients can easily overwhelm the DNs and likely cause OOMs or
> excessive GC.
> The NN should redirect the client to a rack-local location to distribute the
> webhdfs load across multiple hosts. The rack can then use the lightweight
> streamer protocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
