[
https://issues.apache.org/jira/browse/HDFS-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123173#comment-14123173
]
Chris Nauroth commented on HDFS-6967:
-------------------------------------
This is another case where it's a real shame that we're stuck on Jetty 6.
Starting in Jetty 7, it's possible to limit the queue that backs the
{{QueuedThreadPool}}. Here is the relevant constructor in current Jetty 9:
http://download.eclipse.org/jetty/stable-9/apidocs/org/eclipse/jetty/util/thread/QueuedThreadPool.html#QueuedThreadPool(int,
int, int, java.util.concurrent.BlockingQueue)
[~daryn], I know this has always been a touchy subject, but do you think this
problem warrants reopening discussion of a Jetty upgrade? I believe that would
set us up for a relatively simple fix in {{HttpServer2#initializeWebServer}},
perhaps with some new config properties to control queue size.
> DNs may OOM under high webhdfs load
> -----------------------------------
>
> Key: HDFS-6967
> URL: https://issues.apache.org/jira/browse/HDFS-6967
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, webhdfs
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Eric Payne
>
> Webhdfs uses jetty. The size of the request thread pool is limited, but
> jetty will accept and queue infinite connections. Every queued connection is
> "heavy" with buffers, etc. Unlike data streamer connections, thousands of
> webhdfs connections will quickly OOM a DN. The accepted requests must be
> bounded and excess clients rejected so they retry on a new DN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
