[
https://issues.apache.org/jira/browse/HDFS-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16869036#comment-16869036
]
Íñigo Goiri commented on HDFS-14589:
------------------------------------
Not sure if we should use similat to the cost-based RPC FairCallQueue in
HDFS-14403 as the requests might all be the same size but it is worth
experimenting.
> RPC fairness for Datanode data transfers
> ----------------------------------------
>
> Key: HDFS-14589
> URL: https://issues.apache.org/jira/browse/HDFS-14589
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Íñigo Goiri
> Assignee: Xue Liu
> Priority: Major
>
> Currently, the Datanode just replies to the data transfers from the clients
> as soon as they come.
> Eventually, when the {{DataXceiverServer}} runs out of threads, it just
> refuses:
> {code}
> // Make sure the xceiver count is not exceeded
> int curXceiverCount = datanode.getXceiverCount();
> if (curXceiverCount > maxXceiverCount) {
> throw new IOException("Xceiver count " + curXceiverCount
> + " exceeds the limit of concurrent xcievers: "
> + maxXceiverCount);
> }
> {code}
> We had a situation where a user had many containers accessing the same block
> and ending up saturating the 3 Datanodes and messing with the other users.
> Ideally, the Namenode should manage this situation some degree but we can
> still get into this situation.
> We should have some smart in the DN to track this and apply some fairness to
> the number of requests per user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
