[
https://issues.apache.org/jira/browse/HDFS-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433687#comment-17433687
]
Janus Chow commented on HDFS-16275:
-----------------------------------
[~ayushtkn] Thank you for your quick explanation.
In fact, I was curious and confused about the default "false" here. I thought
the considerLoad is as same as the current conditions in `isGoodDatanode`, as
exclude stale or exclude slow node, they are kind of no difference for locality
or non-locality.
For configuration, since the considerLoad is from the config of
"dfs.namenode.replication.considerLoad", would
"dfs.namenode.replication.locality.considerLoad" be a choice?
> [HDFS] Enable considerLoad for localWrite
> -----------------------------------------
>
> Key: HDFS-16275
> URL: https://issues.apache.org/jira/browse/HDFS-16275
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Janus Chow
> Assignee: Janus Chow
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently when client is on the same machine of a datanode, it will try to
> write to the local machine regardless of the load of the datanode, that is
> the xceiverCount.
> In our production cluster, datanode and Nodemanager are running on the same
> server, so when there are heavy jobs running on a labeled queue, the
> corresponding datanodes will have higher xceiverCounts than other datanodes.
> When other clients are trying to write, the exception of "could only be
> replicated to 0 nodes" would be thrown.
> This ticket is to enable considerLoad to avoid the hot local write.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
