Hey Ravi Here are my settings: dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold = 21474836480 (20G) dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction = 0.85f
Chen On Wed, Feb 11, 2015 at 4:36 PM, Ravi Prakash <[email protected]> wrote: > Hi Chen! > > Are you running the balancer? What are you setting > dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold > > > dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction > to? > > > > > On Wednesday, February 11, 2015 7:44 AM, Chen Song < > [email protected]> wrote: > > > We have a hadoop cluster consisting of 500 nodes. But the nodes are not > uniform in term of disk spaces. Half of the racks are newer with 11 volumes > of 1.1T on each node, while the other half have 5 volume of 900GB on each > node. > > dfs.datanode.fsdataset.volume.choosing.policy is set to > org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy. > > It winds up with the state of half of nodes are full while the other half > underutilized. I am wondering if there is a known solution for this problem. > > Thank you for any suggestions. > > -- > Chen Song > > > > -- Chen Song
