Deepak Chander created HDFS-11191:
-------------------------------------
Summary: In the command “hdfs dfsadmin -report” The Configured
Capacity is misleading if the dfs.datanode.data.dir is configured with two
directories from the same file system.
Key: HDFS-11191
URL: https://issues.apache.org/jira/browse/HDFS-11191
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 2.5.0
Environment: SLES 11SP3
HDP 2.5.0
Reporter: Deepak Chander
In the command “hdfs dfsadmin -report” The Configured Capacity is misleading if
the dfs.datanode.data.dir is configured with two directories from the same file
system.
hdfs@kimtest1:~> hdfs dfsadmin -report
Configured Capacity: 239942369274 (223.46 GB)
Present Capacity: 207894724602 (193.62 GB)
DFS Remaining: 207894552570 (193.62 GB)
DFS Used: 172032 (168 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9528000512 (8.87 GB)
DFS Remaining: 70452731902 (65.61 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 13010952192 (12.12 GB)
DFS Remaining: 66969780222 (62.37 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9508691968 (8.86 GB)
DFS Remaining: 70472040446 (65.63 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
If you see my datanode root file system size its only 38GB
kimtest3:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /
kimtest4:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 4.2G 32G 12% /
kimtest2:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /
The below is from hdfs-site.xml file
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value>
</property>
I have removed the other directory grid1 and restarted datanode process.
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn</value>
</property>
Now the size is reflecting correctly
hdfs@kimtest1:/grid> hdfs dfsadmin -report
Configured Capacity: 119971184637 (111.73 GB)
Present Capacity: 103947243517 (96.81 GB)
DFS Remaining: 103947157501 (96.81 GB)
DFS Used: 86016 (84 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4764057600 (4.44 GB)
DFS Remaining: 35226308607 (32.81 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016
Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 6505525248 (6.06 GB)
DFS Remaining: 33484840959 (31.19 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:01 PST 2016
Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4754358272 (4.43 GB)
DFS Remaining: 35236007935 (32.82 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
