Íñigo Goiri created HDFS-14589:
----------------------------------
Summary: RPC fairness for Datanode data transfers
Key: HDFS-14589
URL: https://issues.apache.org/jira/browse/HDFS-14589
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Íñigo Goiri
Assignee: Xue Liu
Currently, the Datanode just replies to the data transfers from the clients as
soon as they come.
Eventually, when the {{DataXceiverServer}} runs out of threads, it just refuses:
{code}
// Make sure the xceiver count is not exceeded
int curXceiverCount = datanode.getXceiverCount();
if (curXceiverCount > maxXceiverCount) {
throw new IOException("Xceiver count " + curXceiverCount
+ " exceeds the limit of concurrent xcievers: "
+ maxXceiverCount);
}
{code}
We had a situation where a user had many containers accessing the same block
and ending up saturating the 3 Datanodes and messing with the other users.
Ideally, the Namenode should manage this situation some degree but we can still
get into this situation.
We should have some smart in the DN to track this and apply some fairness to
the number of requests per user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
