[
https://issues.apache.org/jira/browse/HDFS-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-5639:
--------------------------
Attachment: HDMIT-5092.patch
The patch borrows lots of work from
https://issues.apache.org/jira/i#browse/HBASE-8884 and
https://issues.apache.org/jira/i#browse/HBASE-9461. It improves couple things
specific to hadoop.
1. Scheduler could be a global object that can be shared among different rpc
servers. This is useful in the NN case where there could be two RPC servers;
one for client requests and one for service requests. Currently there is no way
to priortize requests between client RPC and DN RPC. The patch includes both
the new rpc scheduler API in hadoop-common-project and NN's usage of this API.
NN's default RPC scheduler takes care of the scenario where NN uses client RPC
server and service RPC server. New scheduler can be plugged in via config
dfs.namenode.rpc.scheduler.factory.class.
2. This can also be useful in the case of YARN RM where several RPC severs are
used; for example it can prioritize AM RPCs over some client RPCs. The default
behavior for YARN is still one scheduler for RPC server unless it changes to
use a global rpc scheduler.
3. There shouldn't be any change in terms of how RPC scheduling is done for any
hadoop services.
4. Fix the handling of queueSizePerHandler when a specific value is passed in.
The fix is "maxQueueSize = handlerCount * queueSizePerHandler."
5. Update RPCCallBenchmark to support the external rpc scheduler; include a
test RpcScheduler implementation.
6. Move CallQueueLength metric from RPCMetrics to DefaultRpcSchedulerMetrics.
> rpc scheduler abstraction
> -------------------------
>
> Key: HDFS-5639
> URL: https://issues.apache.org/jira/browse/HDFS-5639
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Ming Ma
> Attachments: HDMIT-5092.patch
>
>
> We have run into various issues in namenode and hbase w.r.t. rpc handling in
> multi-tenant clusters. The examples are
> https://issues.apache.org/jira/i#browse/HADOOP-9640
> https://issues.apache.org/jira/i#browse/HBASE-8836
> There are different ideas on how to prioritize rpc requests. It could be
> based on user id, or whether it is read request or write request, or it could
> use specific rule like datanode's RPC is more important than client RPC.
> We want to enable people to implement and experiiment different rpc
> schedulers.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
