[
https://issues.apache.org/jira/browse/HDFS-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17433678#comment-17433678
]
Ayush Saxena commented on HDFS-16275:
-------------------------------------
Ohhk, By any chance have you explored AvailableSpaceBlockPlacementPolicy. That
has a optimisation available for local node as well in form of a config
{{dfs.namenode.available-space-block-placement-policy.balance-local-node}}
I haven't gone through the code, but the change proposed should be configurable
& by default turned off, for backward compatibility
> [HDFS] Enable considerLoad for localWrite
> -----------------------------------------
>
> Key: HDFS-16275
> URL: https://issues.apache.org/jira/browse/HDFS-16275
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Janus Chow
> Assignee: Janus Chow
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently when client is on the same machine of a datanode, it will try to
> write to the local machine regardless of the load of the datanode, that is
> the xceiverCount.
> In our production cluster, datanode and Nodemanager are running on the same
> server, so when there are heavy jobs running on a labeled queue, the
> corresponding datanodes will have higher xceiverCounts than other datanodes.
> When other clients are trying to write, the exception of "could only be
> replicated to 0 nodes" would be thrown.
> This ticket is to enable considerLoad to avoid the hot local write.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
