[
https://issues.apache.org/jira/browse/HADOOP-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Li updated HADOOP-9640:
-----------------------------
Labels: hdfs qos rpc (was: )
Target Version/s: 3.0.0
Affects Version/s: 3.0.0
2.2.0
Release Note: Enable optional RPC-level priority to combat congestion
and make request latencies more consistent.
Status: Patch Available (was: Open)
h1. Effect
In the current released version, the latency increases linearly with increased
CallQueueLength from congestion, until the namenode becomes unusable. With the
FairCallQueue, the latency stays constant, ensuring consistent response times
under varying loads.
h1. Overview
Replaces the _LinkedBlockingQueue_ with a new _FairCallQueue_ class that
contains a configurable number of subqueues. Example:
* Queue 0: Realtime Queue (Optionally reserved for heartbeats/block updates)
* Queue 1: High priority
* Queue 2: Normal priority
* Queue 3: Low priority
h1. Scheduling
RPC calls are placed into an appropriate queue via the _HistoryBasedScheduler_,
which looks at the past 1000 requests to determine the priority of the call.
Example:
* A user who made 90% of the calls will be put into the Low priority queue.
* A user who only made 1% of the calls will be put into High priority. Etc.
h1. Multiplexing
RPC calls are withdrawn from an appropriate queue in a weighted round-robin
fashion. Most of the time, handler threads will prefer to start reading
starting from Queue 0, but occasionally will start reading from Queues 1, 2,
and 3. This is to prevent starvation.
> RPC Congestion Control
> ----------------------
>
> Key: HADOOP-9640
> URL: https://issues.apache.org/jira/browse/HADOOP-9640
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 2.2.0, 3.0.0
> Reporter: Xiaobo Peng
> Labels: hdfs, qos, rpc
> Attachments: rpc-congestion-control-draft-plan.pdf
>
>
> Several production Hadoop cluster incidents occurred where the Namenode was
> overloaded and failed to be responsive. This task is to improve the system
> to detect RPC congestion early, and to provide good diagnostic information
> for alerts that identify suspicious jobs/users so as to restore services
> quickly.
> Excerpted from the communication of one incident, “The map task of a user was
> creating huge number of small files in the user directory. Due to the heavy
> load on NN, the JT also was unable to communicate with NN...The cluster
> became responsive only once the job was killed.”
> Excerpted from the communication of another incident, “Namenode was
> overloaded by GetBlockLocation requests (Correction: should be getFileInfo
> requests. the job had a bug that called getFileInfo for a nonexistent file in
> an endless loop). All other requests to namenode were also affected by this
> and hence all jobs slowed down. Cluster almost came to a grinding
> halt…Eventually killed jobtracker to kill all jobs that are running.”
> Excerpted from HDFS-945, “We've seen defective applications cause havoc on
> the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories
> (60k files) etc.”
--
This message was sent by Atlassian JIRA
(v6.1#6144)
