Good question. I didn't pick up on the fact that fsck disagrees with dfsadmin. Have you tried a full restart? Maybe somebody's information is out of date?
-Joey On Fri, Jun 10, 2011 at 6:22 PM, Robert J Berger <rber...@runa.com> wrote: > I think the files may have been corrupted when I had initially shut down the > node that was still in decommisiioning mode > > Unfortunately I hadn't done the dfsadmin -report any time soon before I had > the incident so I can't be sure that they haven't been there for a while. I > always assumed that the fsck command would tell me if there were issues. > > So will running hadoop fsck -move just move the corrupted replicas and leave > the good ones? Will this work even though fsck does not report any corruption? > > On Jun 9, 2011, at 3:20 PM, Joey Echeverria wrote: > >> hadoop fsck -move will move the corrupt files to /lost+found, which >> will "fix" the report. >> >> Do you know what created the corrupt files? >> >> -Joey >> >> On Thu, Jun 9, 2011 at 3:04 PM, Robert J Berger <rber...@runa.com> wrote: >>> I'm still having this problem and am kind of paralyzed until I figure out >>> how to eliminate these Blocks with corrupt replicas. >>> >>> Here is the output of dfsadmin -report and fsck: >>> >>> dfsadmin -report >>> Configured Capacity: 13723995700736 (12.48 TB) >>> Present Capacity: 13731775356416 (12.49 TB) >>> DFS Remaining: 4079794918277 (3.71 TB) >>> DFS Used: 9651980438139 (8.78 TB) >>> DFS Used%: 70.29% >>> Under replicated blocks: 18 >>> Blocks with corrupt replicas: 34 >>> Missing blocks: 0 >>> >>> ------------------------------------------------- >>> Datanodes available: 9 (9 total, 0 dead) >>> (Not showing the nodes other than the one with Decommission in progress) >>> ... >>> Name: 10.195.10.175:50010 >>> Decommission Status : Decommission in progress >>> Configured Capacity: 1731946381312 (1.58 TB) >>> DFS Used: 1083853885440 (1009.42 GB) >>> Non DFS Used: 0 (0 KB) >>> DFS Remaining: 651169222656(606.45 GB) >>> DFS Used%: 62.58% >>> DFS Remaining%: 37.6% >>> Last contact: Wed Jun 08 18:56:54 UTC 2011 >>> ... >>> >>> And the good bits from fsck: >>> >>> Status: HEALTHY >>> Total size: 2832555958232 B (Total open files size: 134217728 B) >>> Total dirs: 72151 >>> Total files: 65449 (Files currently being written: 9) >>> Total blocks (validated): 95076 (avg. block size 29792544 B) (Total >>> open file blocks (not validated): 10) >>> Minimally replicated blocks: 95076 (100.0 %) >>> Over-replicated blocks: 35667 (37.5142 %) >>> Under-replicated blocks: 18 (0.018932223 %) >>> Mis-replicated blocks: 0 (0.0 %) >>> Default replication factor: 3 >>> Average block replication: 3.376278 >>> Corrupt blocks: 0 >>> Missing replicas: 18 (0.0056074243 %) >>> Number of data-nodes: 9 >>> Number of racks: 1 >>> >>> >>> The filesystem under path '/' is HEALTHY >>> >>> >>> >>> On Jun 8, 2011, at 10:38 AM, Robert J Berger wrote: >>> >>>> Synopsis: >>>> * After shutting down a datanode in a cluster, fsck declares CORRUPT with >>>> missing blocks, >>>> * I restore/restart the datanode and fsck soon declares things healthy >>>> * But dfsadmin -report says a small number of blocks have corrupt replicas >>>> and an even smaller number of under replicated blocks >>>> * After a couple of days that number corrupt replicas and under replicated >>>> blocks stays the same >>>> >>>> Full Story: >>>> My Goal is to rebalance blocks across 3 drives each within 2 datanodes in >>>> a 9 datanode (Replication=3) cluster running hadoop 0.20.1 >>>> (EBS Volumes were added to the datanodes over time so one disk had 95% >>>> usage and the others had significantly less) >>>> >>>> The plan was to decommission the nodes and then wipe the disks and then >>>> add them back in to the cluster. >>>> >>>> Before I started I ran fsck and all was healthy. (Unfortunately I did not >>>> really look at the dfsadmin -report at that time, so I can't be sure if >>>> there were no blocks with corrupt replicas at this point) >>>> >>>> I put two nodes into the Decommission process and after waiting about 36 >>>> hours it hadn't finished decommissioning ether. So I decided to throw >>>> caution to the wind and shut down one of them. (and had taken the node I >>>> was shutting down out of the dfs.exclude.file file, also removed the 2nd >>>> node from the dfs.exclude.file , dfsadmin -refreshNodes but kept the 2nd >>>> node live) >>>> >>>> After shutting down one node, running fsck showed about 400 blocks as >>>> missing. >>>> >>>> So I brought back up the shutdown node (it took a while as I had to >>>> restore it from EBS snapshot) and fsck quickly went back to healthy but >>>> with a significant amount of Over replicated blocks >>>> >>>> I put that node back into the decommissioning state (put just that one >>>> node back in the dfs.exclude.file and ran dfsadmin -refreshNodes. >>>> >>>> After another day or so, its still in the decommissioning mode. Fsck says >>>> the cluster is healthy but still 37% over-replicated blocks. >>>> >>>> But the thing that concerns me is that dfsadmin -report says: >>>> >>>> Under replicated blocks: 18 >>>> Blocks with corrupt replicas: 34 >>>> >>>> So really two questions: >>>> >>>> * Is there a way to force these corrupt replicas and under replicated >>>> blocks to get fixed? >>>> * Is there a way to speed up the decommissioning process (without >>>> restarting the cluster) >>>> >>>> I presume that its not safe for me to take down this node until the >>>> decommissioning completes and/or the corrupt replicas are fixed.. >>>> >>>> And finally, is there a better way to accomplish the original task of >>>> rebalancing disks on a datanode? >>>> >>>> Thanks! >>>> Rob >>>> __________________ >>>> Robert J Berger - CTO >>>> Runa Inc. >>>> +1 408-838-8896 >>>> http://blog.ibd.com >>>> >>>> >>>> >>> >>> __________________ >>> Robert J Berger - CTO >>> Runa Inc. >>> +1 408-838-8896 >>> http://blog.ibd.com >>> >>> >>> >>> >> >> >> >> -- >> Joseph Echeverria >> Cloudera, Inc. >> 443.305.9434 > > __________________ > Robert J Berger - CTO > Runa Inc. > +1 408-838-8896 > http://blog.ibd.com > > > > -- Joseph Echeverria Cloudera, Inc. 443.305.9434
