[
https://issues.apache.org/jira/browse/HADOOP-9640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977949#comment-13977949
]
Ming Ma commented on HADOOP-9640:
---------------------------------
Nice work. Some high-level comments,
1. At some point, we might need to prioritize DN RPC over client RPC so that no
matter what application do to NN RPC and FSNamesystem's global lock, DN's
requests will be processed timely. We can do it in two ways. a) config a global
RPC server and have the pluggable CallQueue handle that. b) have one RPC server
for client and one RPC server of service request, for that we will need some
abstraction like https://issues.apache.org/jira/browse/HDFS-5639.
2. CallQueue priority policy. Perhaps this could leave it to the plugin
implementation. It can be somewhat soft policy like FaiCallQueue, or with some
sort of allocation quota like other schedulers, .e.g., if we know the cluster
has allocated 50% to some group at YARN layer, perhaps it is ok to assume that
NN RPC request for that group can be around 50%.
> RPC Congestion Control with FairCallQueue
> -----------------------------------------
>
> Key: HADOOP-9640
> URL: https://issues.apache.org/jira/browse/HADOOP-9640
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 3.0.0, 2.2.0
> Reporter: Xiaobo Peng
> Assignee: Chris Li
> Labels: hdfs, qos, rpc
> Attachments: MinorityMajorityPerformance.pdf,
> NN-denial-of-service-updated-plan.pdf, faircallqueue.patch,
> faircallqueue2.patch, faircallqueue3.patch, faircallqueue4.patch,
> faircallqueue5.patch, faircallqueue6.patch,
> faircallqueue7_with_runtime_swapping.patch,
> rpc-congestion-control-draft-plan.pdf
>
>
> Several production Hadoop cluster incidents occurred where the Namenode was
> overloaded and failed to respond.
> We can improve quality of service for users during namenode peak loads by
> replacing the FIFO call queue with a [Fair Call
> Queue|https://issues.apache.org/jira/secure/attachment/12616864/NN-denial-of-service-updated-plan.pdf].
> (this plan supersedes rpc-congestion-control-draft-plan).
> Excerpted from the communication of one incident, “The map task of a user was
> creating huge number of small files in the user directory. Due to the heavy
> load on NN, the JT also was unable to communicate with NN...The cluster
> became responsive only once the job was killed.”
> Excerpted from the communication of another incident, “Namenode was
> overloaded by GetBlockLocation requests (Correction: should be getFileInfo
> requests. the job had a bug that called getFileInfo for a nonexistent file in
> an endless loop). All other requests to namenode were also affected by this
> and hence all jobs slowed down. Cluster almost came to a grinding
> halt…Eventually killed jobtracker to kill all jobs that are running.”
> Excerpted from HDFS-945, “We've seen defective applications cause havoc on
> the NameNode, for e.g. by doing 100k+ 'listStatus' on very large directories
> (60k files) etc.”
--
This message was sent by Atlassian JIRA
(v6.2#6252)
