Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 50G 12G 39G 23% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.4G 15G 9% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda2 494M 123M 372M 25% /boot
/dev/mapper/centos-home 2.7T 33M 2.7T 1% /home
That’s from one datanode. The second one is nearly identical. I discovered that
50GB is actually a default. That seems really weird. Disk space is cheap. Why
would you not just use most of the disk and why is it so hard to reset the
default?
Adaryl "Bob" Wakefield, MBA
Principal
Mass Street Analytics, LLC
913.938.6685
www.linkedin.com/in/bobwakefieldmba
Twitter: @BobLovesData
From: Chris Nauroth
Sent: Wednesday, November 04, 2015 12:16 PM
To: [email protected]
Subject: Re: hadoop not using whole disk for HDFS
How are those drives partitioned? Is it possible that the directories pointed
to by the dfs.datanode.data.dir property in hdfs-site.xml reside on partitions
that are sized to only 100 GB? Running commands like df would be a good way to
check this at the OS level, independently of Hadoop.
--Chris Nauroth
From: MBA <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Tuesday, November 3, 2015 at 11:16 AM
To: "[email protected]" <[email protected]>
Subject: Re: hadoop not using whole disk for HDFS
Yeah. It has the current value of 1073741824 which is like 1.07 gig.
B.
From: Chris Nauroth
Sent: Tuesday, November 03, 2015 11:57 AM
To: [email protected]
Subject: Re: hadoop not using whole disk for HDFS
Hi Bob,
Does the hdfs-site.xml configuration file contain the property
dfs.datanode.du.reserved? If this is defined, then the DataNode intentionally
will not use this space for storage of replicas.
<property>
<name>dfs.datanode.du.reserved</name>
<value>0</value>
<description>Reserved space in bytes per volume. Always leave this much space
free for non dfs use.
</description>
</property>
--Chris Nauroth
From: MBA <[email protected]>
Reply-To: "[email protected]" <[email protected]>
Date: Tuesday, November 3, 2015 at 10:51 AM
To: "[email protected]" <[email protected]>
Subject: hadoop not using whole disk for HDFS
I’ve got the Hortonworks distro running on a three node cluster. For some
reason the disk available for HDFS is MUCH less than the total disk space. Both
of my data nodes have 3TB hard drives. Only 100GB of that is being used for
HDFS. Is it possible that I have a setting wrong somewhere?
B.
