Good to know, thanks for the update! On Wed, Sep 19, 2012 at 8:29 PM, Artem Ervits <[email protected]> wrote: > Harsh, > > That was actually what it was. I was messing with HBase install and edited > the /etc/hosts file to add the 127.0.0.1 address. Once I removed the entry, > datanodes were able to see the namenode. I was also able to successfully test > the recovery. Hadoop fsck -blocks reports a healthy filesystem now. > > Thank you very much. > > -----Original Message----- > From: Harsh J [mailto:[email protected]] > Sent: Tuesday, September 18, 2012 11:37 PM > To: [email protected] > Subject: Re: Hadoop recovery test > > Artem, > > If you check the logs of the other DNs, do you see issues with connectivity > to NameNode? > > Basic questions, but need to ask to be sure: have you checked if the > firewalls are down or properly configured? Are you sure that your hostname of > the master machine resolves not to the loopback address but to the external > interface provided IP? > > On Tue, Sep 18, 2012 at 10:29 PM, Artem Ervits <[email protected]> wrote: >> I didn't realize that I didn't edit core-site and mapred-site on all >> machines to point to the new namenode. Although that didn't make a >> difference, I still see only one datanode which Is also the namenode: >> >> Datanodes available: 1 (1 total, 0 dead) >> >> Name: 127.0.0.1:50010 >> Decommission Status : Normal >> Configured Capacity: 105425190912 (98.18 GB) DFS Used: 1058557952 >> (1009.52 MB) Non DFS Used: 200396800 (191.11 MB) DFS Remaining: >> 104166236160(97.01 GB) DFS Used%: 1% DFS Remaining%: 98.81% Last >> contact: Tue Sep 18 12:58:07 EDT 2012 >> >> The other strange thing is that it points to local 127.0.0.1 rather than >> namenode's IP. >> >> -----Original Message----- >> From: Artem Ervits [mailto:[email protected]] >> Sent: Tuesday, September 18, 2012 9:57 AM >> To: [email protected] >> Cc: James Brown >> Subject: RE: Hadoop recovery test >> >> No it only sees itself. It doesn't see the rest of the nodes. >> >> -----Original Message----- >> From: James Brown [mailto:[email protected]] >> Sent: Monday, September 17, 2012 5:49 PM >> To: [email protected] >> Subject: Re: Hadoop recovery test >> >> Does the new NameNode server see all of the DataNodes? >> >> On 9/17/2012 2:38 PM, Artem Ervits wrote: >>> Hello all, >>> >>> I am testing the Hadoop recovery as per >>> http://wiki.apache.org/hadoop/NameNode document. But instead of using >>> an NFS share, I am copying to another directory. Then when I shut >>> down the cluster, I scp that directory to another server and start >>> Hadoop cluster using that machine as the namenode. I see in the log >>> that some blocks are corrupt and/or missing. Do I have to wait for >>> replication to recover all blocks or am I doing something else >>> altogether? I am using Hadoop 1.0.3. Can someone point me to a more >>> detailed document than the wiki in case I'm doing something wrong. >>> >>> p.s. if I restart the cluster using the original namenode, filesystem >>> reports as healthy. >>> >>> Thank you. >>> >>> . >>> >>> /hdfs/hadoop/tmp/mapred/system/jobtracker.info: CORRUPT block >>> blk_9043419219670949307 >>> >>> /hdfs/hadoop/tmp/mapred/system/jobtracker.info: MISSING 1 blocks of >>> total size 4 B... >>> >>> /user/hduser/teragen/_logs/history/job_201209120941_0002_1347458152167_hduser_TeraGen: >>> Under replicated blk_-976282286234272458_1079. Target Replicas is 3 >>> but found 1 replica(s). >>> >>> . >>> >>> /user/hduser/teragen/_logs/history/job_201209120941_0002_conf.xml: >>> Under replicated blk_137658109390447967_1075. Target Replicas is 3 >>> but found 1 replica(s). >>> >>> . >>> >>> /user/hduser/teragen/_partition.lst: Under replicated >>> blk_-3005280481530403302_1080. Target Replicas is 3 but found 1 replica(s). >>> >>> . >>> >>> /user/hduser/teragen/part-00000: Under replicated >>> blk_-7008813028808832816_1077. Target Replicas is 3 but found 1 replica(s). >>> >>> . >>> >>> /user/hduser/teragen/part-00001: Under replicated >>> blk_-5256967771026054061_1078. Target Replicas is 3 but found 1 replica(s). >>> >>> .. >>> >>> /user/hduser/teragen-out/_logs/history/job_201209120941_0003_1347458249920_hduser_TeraSort: >>> Under replicated blk_1137779303840586677_1089. Target Replicas is 3 >>> but found 1 replica(s). >>> >>> . >>> >>> /user/hduser/teragen-out/_logs/history/job_201209120941_0003_conf.xml: >>> Under replicated blk_7701720691642589882_1086. Target Replicas is 3 >>> but found 1 replica(s). >>> >>> . >>> >>> /user/hduser/teragen-out/part-00000: CORRUPT block >>> blk_8059469267617478950 >>> >>> /user/hduser/teragen-out/part-00000: MISSING 1 blocks of total size >>> 1000000 B... >>> >>> /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_1347458495941_hduser_TeraValidate: >>> Under replicated blk_5680565744062298575_1098. Target Replicas is 3 >>> but found 1 replica(s). >>> >>> . >>> >>> /user/hduser/teragen-validate/_logs/history/job_201209120941_0004_conf.xml: >>> Under replicated blk_1566253937037013126_1095. Target Replicas is 3 >>> but found 1 replica(s). >>> >>> .Status: CORRUPT >>> >>> Total size: 1050720258 B >>> >>> Total dirs: 39 >>> >>> Total files: 32 >>> >>> Total blocks (validated): 42 (avg. block size 25017149 B) >>> >>> ******************************** >>> >>> CORRUPT FILES: 2 >>> >>> MISSING BLOCKS: 2 >>> >>> MISSING SIZE: 1000004 B >>> >>> CORRUPT BLOCKS: 2 >>> >>> ******************************** >>> >>> Minimally replicated blocks: 40 (95.2381 %) >>> >>> Over-replicated blocks: 0 (0.0 %) >>> >>> Under-replicated blocks: 40 (95.2381 %) >>> >>> Mis-replicated blocks: 0 (0.0 %) >>> >>> Default replication factor: 3 >>> >>> Average block replication: 0.95238096 >>> >>> Corrupt blocks: 2 >>> >>> Missing replicas: 80 (200.0 %) >>> >>> Number of data-nodes: 1 >>> >>> Number of racks: 1 >>> >>> FSCK ended at Mon Sep 17 17:29:08 EDT 2012 in 21 milliseconds >>> >>> The filesystem under path '/' is CORRUPT >>> >>> Artem Ervits >>> >>> Data Analyst >>> >>> New York Presbyterian Hospital >>> >>> >>> --------------------------------------------------------------------- >>> - >>> -- This electronic message is intended to be for the use only of the >>> named recipient, and may contain information that is confidential or >>> privileged. If you are not the intended recipient, you are hereby >>> notified that any disclosure, copying, distribution or use of the >>> contents of this message is strictly prohibited. If you have received >>> this message in error or are not the named recipient, please notify >>> us immediately by contacting the sender at the electronic mail >>> address noted above, and delete and destroy all copies of this message. >>> Thank you. >>> >>> -------------------- >>> >>> This electronic message is intended to be for the use only of the named >>> recipient, and may contain information that is confidential or privileged. >>> If you are not the intended recipient, you are hereby notified that any >>> disclosure, copying, distribution or use of the contents of this message is >>> strictly prohibited. If you have received this message in error or are not >>> the named recipient, please notify us immediately by contacting the sender >>> at the electronic mail address noted above, and delete and destroy all >>> copies of this message. Thank you. >>> >>> -------------------- >>> >>> This electronic message is intended to be for the use only of the named >>> recipient, and may contain information that is confidential or privileged. >>> If you are not the intended recipient, you are hereby notified that any >>> disclosure, copying, distribution or use of the contents of this message is >>> strictly prohibited. If you have received this message in error or are not >>> the named recipient, please notify us immediately by contacting the sender >>> at the electronic mail address noted above, and delete and destroy all >>> copies of this message. Thank you. >>> >>> >> >> >> >> -------------------- >> >> This electronic message is intended to be for the use only of the named >> recipient, and may contain information that is confidential or privileged. >> If you are not the intended recipient, you are hereby notified that any >> disclosure, copying, distribution or use of the contents of this message is >> strictly prohibited. If you have received this message in error or are not >> the named recipient, please notify us immediately by contacting the sender >> at the electronic mail address noted above, and delete and destroy all >> copies of this message. Thank you. >> >> >> >> >> -------------------- >> >> This electronic message is intended to be for the use only of the named >> recipient, and may contain information that is confidential or privileged. >> If you are not the intended recipient, you are hereby notified that any >> disclosure, copying, distribution or use of the contents of this message is >> strictly prohibited. If you have received this message in error or are not >> the named recipient, please notify us immediately by contacting the sender >> at the electronic mail address noted above, and delete and destroy all >> copies of this message. Thank you. >> >> >> >> >> ________________________________ >> >> Confidential Information subject to NYP's (and its affiliates') information >> management and security policies (http://infonet.nyp.org/QA/HospitalManual). >> >> >> -------------------- >> >> This electronic message is intended to be for the use only of the named >> recipient, and may contain information that is confidential or privileged. >> If you are not the intended recipient, you are hereby notified that any >> disclosure, copying, distribution or use of the contents of this message is >> strictly prohibited. If you have received this message in error or are not >> the named recipient, please notify us immediately by contacting the sender >> at the electronic mail address noted above, and delete and destroy all >> copies of this message. Thank you. >> >> >> >> >> -------------------- >> >> This electronic message is intended to be for the use only of the named >> recipient, and may contain information that is confidential or privileged. >> If you are not the intended recipient, you are hereby notified that any >> disclosure, copying, distribution or use of the contents of this message is >> strictly prohibited. If you have received this message in error or are not >> the named recipient, please notify us immediately by contacting the sender >> at the electronic mail address noted above, and delete and destroy all >> copies of this message. Thank you. >> >> >> > > > > -- > Harsh J > > > -------------------- > > This electronic message is intended to be for the use only of the named > recipient, and may contain information that is confidential or privileged. > If you are not the intended recipient, you are hereby notified that any > disclosure, copying, distribution or use of the contents of this message is > strictly prohibited. If you have received this message in error or are not > the named recipient, please notify us immediately by contacting the sender at > the electronic mail address noted above, and delete and destroy all copies of > this message. Thank you. > > > > > -------------------- > > This electronic message is intended to be for the use only of the named > recipient, and may contain information that is confidential or privileged. > If you are not the intended recipient, you are hereby notified that any > disclosure, copying, distribution or use of the contents of this message is > strictly prohibited. If you have received this message in error or are not > the named recipient, please notify us immediately by contacting the sender at > the electronic mail address noted above, and delete and destroy all copies of > this message. Thank you. > > >
-- Harsh J
