I tried copying two ways. Once while Hadoop was running and second time when I shut down the original cluster. I used scp -r command, is there a better option, rsync? I also tried scp with -rp4 switch but still can't get the folders to look identical
Original node: total 20 drwxr-xr-x. 16 hduser hadoop 4096 Sep 17 13:31 .. drwxrwx---. 2 hduser hadoop 4096 Sep 17 13:41 image drwxrwx---. 2 hduser hadoop 4096 Sep 17 16:39 previous.checkpoint drwxrwx---. 2 hduser hadoop 4096 Sep 18 09:57 current drwxrwx---. 5 hduser hadoop 4096 Sep 18 09:59 . new namenode: total 20 drwxrwx---. 2 hduser hadoop 4096 Sep 17 13:41 image drwxrwx---. 2 hduser hadoop 4096 Sep 17 16:39 previous.checkpoint drwxrwx---. 2 hduser hadoop 4096 Sep 18 09:57 current drwxrwx---. 5 hduser hadoop 4096 Sep 18 09:59 . drwxr-xr-x. 17 hduser hadoop 4096 Sep 18 10:03 .. From: Robert Molina [mailto:[email protected]] Sent: Monday, September 17, 2012 5:55 PM To: [email protected] Subject: Re: Hadoop recovery test Hi Artem, At what point do you do the copy, was namenode still running? Does the copy of the edits file and fsimage file match up with the original (i.e filesize)? -Robert On Mon, Sep 17, 2012 at 2:38 PM, Artem Ervits <[email protected]<mailto:[email protected]>> wrote: Hello all, I am testing the Hadoop recovery as per http://wiki.apache.org/hadoop/NameNode document. But instead of using an NFS share, I am copying to another directory. Then when I shut down the cluster, I scp that directory to another server and start Hadoop cluster using that machine as the namenode. I see in the log that some blocks are corrupt and/or missing. Do I have to wait for replication to recover all blocks or am I doing something else altogether? I am using Hadoop 1.0.3. Can someone point me to a more detailed document than the wiki in case I'm doing something wrong. p.s. if I restart the cluster using the original namenode, filesystem reports as healthy. Thank you. . /hdfs/hadoop/tmp/mapred/system/jobtracker.info<http://jobtracker.info>: CORRUPT block blk_9043419219670949307 /hdfs/hadoop/tmp/mapred/system/jobtracker.info<http://jobtracker.info>: MISSING 1 blocks of total size 4 B... /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: Under replicated blk_-976282286234272458_1079. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: Under replicated blk_137658109390447967_1075. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/_partition.lst: Under replicated blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/part-00000: Under replicated blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen/part-00001: Under replicated blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). .. /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: Under replicated blk_1137779303840586677_1089. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: Under replicated blk_7701720691642589882_1086. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-out/part-00000: CORRUPT block blk_8059469267617478950 /user/hduser/teragen-out/part-00000: MISSING 1 blocks of total size 1000000 B... /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_1347458495941_hduser_TeraValidate: Under replicated blk_5680565744062298575_1098. Target Replicas is 3 but found 1 replica(s). . /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_conf.xml: Under replicated blk_1566253937037013126_1095. Target Replicas is 3 but found 1 replica(s). .Status: CORRUPT Total size: 1050720258 B Total dirs: 39 Total files: 32 Total blocks (validated): 42 (avg. block size 25017149 B) ******************************** CORRUPT FILES: 2 MISSING BLOCKS: 2 MISSING SIZE: 1000004 B CORRUPT BLOCKS: 2 ******************************** Minimally replicated blocks: 40 (95.2381 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 40 (95.2381 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 0.95238096 Corrupt blocks: 2 Missing replicas: 80 (200.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Mon Sep 17 17:29:08 EDT 2012 in 21 milliseconds The filesystem under path '/' is CORRUPT Artem Ervits Data Analyst New York Presbyterian Hospital ________________________________ This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. -------------------- This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. -------------------- This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. -------------------- This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. -------------------- This electronic message is intended to be for the use only of the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you.
