[
https://issues.apache.org/jira/browse/HDFS-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepak Chander updated HDFS-11191:
----------------------------------
Description:
In the command “hdfs dfsadmin -report” The Configured Capacity is misleading if
the dfs.datanode.data.dir is configured with two directories from the same file
system.
hdfs@kimtest1:~> hdfs dfsadmin -report
Configured Capacity: 239942369274 (223.46 GB)
Present Capacity: 207894724602 (193.62 GB)
DFS Remaining: 207894552570 (193.62 GB)
DFS Used: 172032 (168 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9528000512 (8.87 GB)
DFS Remaining: 70452731902 (65.61 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 13010952192 (12.12 GB)
DFS Remaining: 66969780222 (62.37 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9508691968 (8.86 GB)
DFS Remaining: 70472040446 (65.63 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
If you see my datanode root file system size its only 38GB
kimtest3:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /
kimtest4:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 4.2G 32G 12% /
kimtest2:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /
The below is from hdfs-site.xml file
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value>
</property>
I have removed the other directory grid1 and restarted datanode process.
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn</value>
</property>
Now the size is reflecting correctly
hdfs@kimtest1:/grid> hdfs dfsadmin -report
Configured Capacity: 119971184637 (111.73 GB)
Present Capacity: 103947243517 (96.81 GB)
DFS Remaining: 103947157501 (96.81 GB)
DFS Used: 86016 (84 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4764057600 (4.44 GB)
DFS Remaining: 35226308607 (32.81 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016
Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 6505525248 (6.06 GB)
DFS Remaining: 33484840959 (31.19 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:01 PST 2016
Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4754358272 (4.43 GB)
DFS Remaining: 35236007935 (32.82 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016
was:
In the command “hdfs dfsadmin -report” The Configured Capacity is misleading if
the dfs.datanode.data.dir is configured with two directories from the same file
system.
hdfs@kimtest1:~> hdfs dfsadmin -report
Configured Capacity: 239942369274 (223.46 GB)
Present Capacity: 207894724602 (193.62 GB)
DFS Remaining: 207894552570 (193.62 GB)
DFS Used: 172032 (168 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9528000512 (8.87 GB)
DFS Remaining: 70452731902 (65.61 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 13010952192 (12.12 GB)
DFS Remaining: 66969780222 (62.37 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 79980789758 (74.49 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 9508691968 (8.86 GB)
DFS Remaining: 70472040446 (65.63 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 06:59:02 PST 2016
If you see my datanode root file system size its only 38GB
kimtest3:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /
kimtest4:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 4.2G 32G 12% /
kimtest2:~ # df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root 38G 2.6G 33G 8% /
The below is from hdfs-site.xml file
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value>
</property>
I have removed the other directory grid1 and restarted datanode process.
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///grid/hadoop/hdfs/dn</value>
</property>
Now the size is reflecting correctly
hdfs@kimtest1:/grid> hdfs dfsadmin -report
Configured Capacity: 119971184637 (111.73 GB)
Present Capacity: 103947243517 (96.81 GB)
DFS Remaining: 103947157501 (96.81 GB)
DFS Used: 86016 (84 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 172.26.79.87:50010 (kimtest3)
Hostname: kimtest3
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4764057600 (4.44 GB)
DFS Remaining: 35226308607 (32.81 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.09%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016
Name: 172.26.80.38:50010 (kimtest4)
Hostname: kimtest4
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 6505525248 (6.06 GB)
DFS Remaining: 33484840959 (31.19 GB)
DFS Used%: 0.00%
DFS Remaining%: 83.73%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:01 PST 2016
Name: 172.26.79.86:50010 (kimtest2)
Hostname: kimtest2
Decommission Status : Normal
Configured Capacity: 39990394879 (37.24 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 4754358272 (4.43 GB)
DFS Remaining: 35236007935 (32.82 GB)
DFS Used%: 0.00%
DFS Remaining%: 88.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Tue Nov 29 07:34:02 PST 2016
If the file system is same for the directories listed under
dfs.datanode.data.dir, we expect to see the original size of the file system.
The code should differentiate between the file systems.
> In the command “hdfs dfsadmin -report” The Configured Capacity is misleading
> if the dfs.datanode.data.dir is configured with two directories from the same
> file system.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-11191
> URL: https://issues.apache.org/jira/browse/HDFS-11191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.5.0
> Environment: SLES 11SP3
> HDP 2.5.0
> Reporter: Deepak Chander
>
> In the command “hdfs dfsadmin -report” The Configured Capacity is misleading
> if the dfs.datanode.data.dir is configured with two directories from the same
> file system.
> hdfs@kimtest1:~> hdfs dfsadmin -report
> Configured Capacity: 239942369274 (223.46 GB)
> Present Capacity: 207894724602 (193.62 GB)
> DFS Remaining: 207894552570 (193.62 GB)
> DFS Used: 172032 (168 KB)
> DFS Used%: 0.00%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> -------------------------------------------------
> Live datanodes (3):
> Name: 172.26.79.87:50010 (kimtest3)
> Hostname: kimtest3
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 9528000512 (8.87 GB)
> DFS Remaining: 70452731902 (65.61 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.09%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> Name: 172.26.80.38:50010 (kimtest4)
> Hostname: kimtest4
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 13010952192 (12.12 GB)
> DFS Remaining: 66969780222 (62.37 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 83.73%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> Name: 172.26.79.86:50010 (kimtest2)
> Hostname: kimtest2
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 9508691968 (8.86 GB)
> DFS Remaining: 70472040446 (65.63 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.11%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> If you see my datanode root file system size its only 38GB
> kimtest3:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 2.6G 33G 8% /
> kimtest4:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 4.2G 32G 12% /
> kimtest2:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 2.6G 33G 8% /
> The below is from hdfs-site.xml file
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value>
> </property>
> I have removed the other directory grid1 and restarted datanode process.
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:///grid/hadoop/hdfs/dn</value>
> </property>
> Now the size is reflecting correctly
> hdfs@kimtest1:/grid> hdfs dfsadmin -report
> Configured Capacity: 119971184637 (111.73 GB)
> Present Capacity: 103947243517 (96.81 GB)
> DFS Remaining: 103947157501 (96.81 GB)
> DFS Used: 86016 (84 KB)
> DFS Used%: 0.00%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> -------------------------------------------------
> Live datanodes (3):
> Name: 172.26.79.87:50010 (kimtest3)
> Hostname: kimtest3
> Decommission Status : Normal
> Configured Capacity: 39990394879 (37.24 GB)
> DFS Used: 28672 (28 KB)
> Non DFS Used: 4764057600 (4.44 GB)
> DFS Remaining: 35226308607 (32.81 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.09%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 07:34:02 PST 2016
> Name: 172.26.80.38:50010 (kimtest4)
> Hostname: kimtest4
> Decommission Status : Normal
> Configured Capacity: 39990394879 (37.24 GB)
> DFS Used: 28672 (28 KB)
> Non DFS Used: 6505525248 (6.06 GB)
> DFS Remaining: 33484840959 (31.19 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 83.73%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 07:34:01 PST 2016
> Name: 172.26.79.86:50010 (kimtest2)
> Hostname: kimtest2
> Decommission Status : Normal
> Configured Capacity: 39990394879 (37.24 GB)
> DFS Used: 28672 (28 KB)
> Non DFS Used: 4754358272 (4.43 GB)
> DFS Remaining: 35236007935 (32.82 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.11%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 07:34:02 PST 2016
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
