[
https://issues.apache.org/jira/browse/HDFS-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710201#comment-15710201
]
Weiwei Yang commented on HDFS-11191:
------------------------------------
At present the datanode capacity is a simple cumulative of each data volume
capacity, so even 2 volumes are from same file system, it gets calculated
twice. To fix this, o.a.h.fs.DF already provides to method to retrieve the file
system, but we don't want to include that in each storage report in the
heartbeat because it is a heavy unix call. I can take a look at this see if
there is a neat way to fix this.
> In the command “hdfs dfsadmin -report” The Configured Capacity is misleading
> if the dfs.datanode.data.dir is configured with two directories from the same
> file system.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-11191
> URL: https://issues.apache.org/jira/browse/HDFS-11191
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.5.0
> Environment: SLES 11SP3
> HDP 2.5.0
> Reporter: Deepak Chander
>
> In the command “hdfs dfsadmin -report” The Configured Capacity is misleading
> if the dfs.datanode.data.dir is configured with two directories from the same
> file system.
> hdfs@kimtest1:~> hdfs dfsadmin -report
> Configured Capacity: 239942369274 (223.46 GB)
> Present Capacity: 207894724602 (193.62 GB)
> DFS Remaining: 207894552570 (193.62 GB)
> DFS Used: 172032 (168 KB)
> DFS Used%: 0.00%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> -------------------------------------------------
> Live datanodes (3):
> Name: 172.26.79.87:50010 (kimtest3)
> Hostname: kimtest3
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 9528000512 (8.87 GB)
> DFS Remaining: 70452731902 (65.61 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.09%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> Name: 172.26.80.38:50010 (kimtest4)
> Hostname: kimtest4
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 13010952192 (12.12 GB)
> DFS Remaining: 66969780222 (62.37 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 83.73%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> Name: 172.26.79.86:50010 (kimtest2)
> Hostname: kimtest2
> Decommission Status : Normal
> Configured Capacity: 79980789758 (74.49 GB)
> DFS Used: 57344 (56 KB)
> Non DFS Used: 9508691968 (8.86 GB)
> DFS Remaining: 70472040446 (65.63 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.11%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 06:59:02 PST 2016
> If you see my datanode root file system size its only 38GB
> kimtest3:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 2.6G 33G 8% /
> kimtest4:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 4.2G 32G 12% /
> kimtest2:~ # df -h /
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/system-root 38G 2.6G 33G 8% /
> The below is from hdfs-site.xml file
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:///grid/hadoop/hdfs/dn, file:///grid1/hadoop/hdfs/dn</value>
> </property>
> I have removed the other directory grid1 and restarted datanode process.
> <property>
> <name>dfs.datanode.data.dir</name>
> <value>file:///grid/hadoop/hdfs/dn</value>
> </property>
> Now the size is reflecting correctly
> hdfs@kimtest1:/grid> hdfs dfsadmin -report
> Configured Capacity: 119971184637 (111.73 GB)
> Present Capacity: 103947243517 (96.81 GB)
> DFS Remaining: 103947157501 (96.81 GB)
> DFS Used: 86016 (84 KB)
> DFS Used%: 0.00%
> Under replicated blocks: 0
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> -------------------------------------------------
> Live datanodes (3):
> Name: 172.26.79.87:50010 (kimtest3)
> Hostname: kimtest3
> Decommission Status : Normal
> Configured Capacity: 39990394879 (37.24 GB)
> DFS Used: 28672 (28 KB)
> Non DFS Used: 4764057600 (4.44 GB)
> DFS Remaining: 35226308607 (32.81 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.09%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 07:34:02 PST 2016
> Name: 172.26.80.38:50010 (kimtest4)
> Hostname: kimtest4
> Decommission Status : Normal
> Configured Capacity: 39990394879 (37.24 GB)
> DFS Used: 28672 (28 KB)
> Non DFS Used: 6505525248 (6.06 GB)
> DFS Remaining: 33484840959 (31.19 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 83.73%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 07:34:01 PST 2016
> Name: 172.26.79.86:50010 (kimtest2)
> Hostname: kimtest2
> Decommission Status : Normal
> Configured Capacity: 39990394879 (37.24 GB)
> DFS Used: 28672 (28 KB)
> Non DFS Used: 4754358272 (4.43 GB)
> DFS Remaining: 35236007935 (32.82 GB)
> DFS Used%: 0.00%
> DFS Remaining%: 88.11%
> Configured Cache Capacity: 0 (0 B)
> Cache Used: 0 (0 B)
> Cache Remaining: 0 (0 B)
> Cache Used%: 100.00%
> Cache Remaining%: 0.00%
> Xceivers: 2
> Last contact: Tue Nov 29 07:34:02 PST 2016
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
