Hi, I think you are referring DFS Used (from NameNode report) and Total size (from fsck) values right?.
*DFS Used:* This contains the total hdfs space used on all the connected data nodes, in your case 230296610816 (214.48 GB). ** *Total Size:* Fsck utility looks for the blocks in namespace , it will check all the blocks one by one including replicated blocks, fsck retrieve all the information from name node only. Therefore fsck total size contains the size of total blocks on hdfs excluding replicas. Hope this will help you. Thanks On Sun, Aug 11, 2013 at 10:44 PM, Yogini Gulkotwar < [email protected]> wrote: > Hi All, > > I have a CDH4 hadoop cluster setup with 3 datanodes and a data replication > factor of 2. > > When I try to check the consumed dfs space, I get different values using > the "hdfs dfsadmin -report" and "hdfs fsck" command. > Could anyone please help me understand the reason behind the discrepancy > in the values? > > I get the following output: > > *# sudo -u hdfs hdfs dfsadmin -report* > > > Configured Capacity: 321252989337600 (292.18 TB) > Present Capacity: 264896108259328 (240.92 TB) > DFS Remaining: 264665811648512 (240.71 TB) > DFS Used: 230296610816 (214.48 GB) > DFS Used%: 0.09% > Under replicated blocks: 19 > Blocks with corrupt replicas: 0 > Missing blocks: 0 > > ------------------------------------------------- > Datanodes available: 3 (3 total, 0 dead) > > Live datanodes: > Name: (slave1) > Hostname: localhost > Decommission Status : Normal > Configured Capacity: 107084329779200 (97.39 TB) > DFS Used: 77728510976 (72.39 GB) > Non DFS Used: 18784664751104 (17.08 TB) > DFS Remaining: 88221936517120 (80.24 TB) > DFS Used%: 0.07% > DFS Remaining%: 82.39% > Last contact: Fri Aug 09 13:26:38 IST 2013 > > > Name: (slave3) > Hostname: localhost > Decommission Status : Normal > Configured Capacity: 107084329779200 (97.39 TB) > DFS Used: 76206287872 (70.97 GB) > Non DFS Used: 18786185925632 (17.09 TB) > DFS Remaining: 88221937565696 (80.24 TB) > DFS Used%: 0.07% > DFS Remaining%: 82.39% > Last contact: Fri Aug 09 13:26:37 IST 2013 > > > Name:(slave2) > Hostname: localhost > Decommission Status : Normal > Configured Capacity: 107084329779200 (97.39 TB) > DFS Used: 76361811968 (71.12 GB) > Non DFS Used: 18786030401536 (17.09 TB) > DFS Remaining: 88221937565696 (80.24 TB) > DFS Used%: 0.07% > DFS Remaining%: 82.39% > > > -------------------------------------------------------------------------------------------------------------------------- > *# sudo -u hdfs hadoop fsck /* > > > Connecting to namenode via http://master1:50070 > > > Status: HEALTHY > Total size: 75245213337 B > Total dirs: 3203 > Total files: 7893 > Total blocks (validated): 7642 (avg. block size 9846272 B) > Minimally replicated blocks: 7642 (100.0 %) > Over-replicated blocks: 0 (0.0 %) > Under-replicated blocks: 19 (0.24862601 %) > Mis-replicated blocks: 0 (0.0 %) > Default replication factor: 2 > Average block replication: 2.0024862 > Corrupt blocks: 0 > Missing replicas: 133 (0.86162215 %) > Number of data-nodes: 3 > Number of racks: 1 > FSCK ended at Fri Aug 09 14:01:47 IST 2013 in 266 milliseconds > > > The filesystem under path '/' is HEALTHY > > > ---------------------------------------------------------------------------------------------------------------------------------------------------- > > > *# sudo -u hdfs hadoop fs -count -q /* > 2147483647 2147472547 none inf 3203 > 7897 75245470999 / > > > > Thanks & Regards, > *Yogini Gulkotwar* > *Flutura Decision Sciences & Analytics, Bangalore* > *Email*: [email protected]<[email protected]> > *Website*: www.fluturasolutions.com > >
