Hi Bob,
Seems like you have configured to disk dir to be other than an folder in /home, if so try creating another folder and add to "dfs.datanode.data.dir" seperated by comma instead of trying to reset the default. And its also advised not to use the root partition "/" to be configured for HDFS data dir, if the Dir usage hits the maximum then OS might fail to function properly. Regards, + Naga ________________________________ From: P lva [[email protected]] Sent: Thursday, November 05, 2015 03:11 To: [email protected] Subject: Re: hadoop not using whole disk for HDFS What does your dfs.datanode.data.dir point to ? On Wed, Nov 4, 2015 at 4:14 PM, Adaryl "Bob" Wakefield, MBA <[email protected]<mailto:[email protected]>> wrote: Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 50G 12G 39G 23% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 0 16G 0% /dev/shm tmpfs 16G 1.4G 15G 9% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/sda2 494M 123M 372M 25% /boot /dev/mapper/centos-home 2.7T 33M 2.7T 1% /home That’s from one datanode. The second one is nearly identical. I discovered that 50GB is actually a default. That seems really weird. Disk space is cheap. Why would you not just use most of the disk and why is it so hard to reset the default? Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685<tel:913.938.6685> www.linkedin.com/in/bobwakefieldmba<http://www.linkedin.com/in/bobwakefieldmba> Twitter: @BobLovesData From: Chris Nauroth<mailto:[email protected]> Sent: Wednesday, November 04, 2015 12:16 PM To: [email protected]<mailto:[email protected]> Subject: Re: hadoop not using whole disk for HDFS How are those drives partitioned? Is it possible that the directories pointed to by the dfs.datanode.data.dir property in hdfs-site.xml reside on partitions that are sized to only 100 GB? Running commands like df would be a good way to check this at the OS level, independently of Hadoop. --Chris Nauroth From: MBA <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, November 3, 2015 at 11:16 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: hadoop not using whole disk for HDFS Yeah. It has the current value of 1073741824 which is like 1.07 gig. B. From: Chris Nauroth<mailto:[email protected]> Sent: Tuesday, November 03, 2015 11:57 AM To: [email protected]<mailto:[email protected]> Subject: Re: hadoop not using whole disk for HDFS Hi Bob, Does the hdfs-site.xml configuration file contain the property dfs.datanode.du.reserved? If this is defined, then the DataNode intentionally will not use this space for storage of replicas. <property> <name>dfs.datanode.du.reserved</name> <value>0</value> <description>Reserved space in bytes per volume. Always leave this much space free for non dfs use. </description> </property> --Chris Nauroth From: MBA <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Tuesday, November 3, 2015 at 10:51 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: hadoop not using whole disk for HDFS I’ve got the Hortonworks distro running on a three node cluster. For some reason the disk available for HDFS is MUCH less than the total disk space. Both of my data nodes have 3TB hard drives. Only 100GB of that is being used for HDFS. Is it possible that I have a setting wrong somewhere? B.
