Hi Chen!
Are you running the balancer? What are you setting
dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold
dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fractionto?
On Wednesday, February 11, 2015 7:44 AM, Chen Song
<[email protected]> wrote:
We have a hadoop cluster consisting of 500 nodes. But the nodes are not
uniform in term of disk spaces. Half of the racks are newer with 11 volumes of
1.1T on each node, while the other half have 5 volume of 900GB on each node.
dfs.datanode.fsdataset.volume.choosing.policy is set to
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy.
It winds up with the state of half of nodes are full while the other half
underutilized. I am wondering if there is a known solution for this problem.
Thank you for any suggestions.
--
Chen Song
