Gents, Need to share with you my embarrassment... Solved this issue.. How?
Well, while following the installation instructions I thought I installed all the daemons, but, after checking the init.d folder I could not find hadoop-hdfs-datanode script so (thinking I acciddentslly deleted it) I merely scp'ied the script from another node. I've tried in vain to start that node for at least 13 hours until, I went on installing hadoop on a new node, I realised that I missed data-node installation all together. I was supposed to run: sudo yum install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode but I ran only sudo yum install hadoop-0.20-mapreduce-tasktracker After installing datanode and reformatting the namespace, datanode started like a new engine. Silly me. Oh well. :) Calm seas do not make good sailors. AK47 From: Kartashov, Andy Sent: Thursday, October 25, 2012 3:40 PM To: [email protected] Subject: datanode daemon Guys, I finally solved ALL the Errors: in ...datanode*.log after trying to start the node with "service datanode start". The errors were: - conflicting NN DD ids - solved through reformatting NN. - could not connect to 127.0.0.1:8020 - Connection refused - solved through correcting a typo inside hdfs-site.xml under dfs.namenode.http-address; somehow had the default value i/o localhost. (Running pseudo-mode) - conf was pointing to the wrong sLink - solved by running alternatives -set hadoop-conf <conf.myconf> However, when I run "service -status-all", still see that datanode [FAILED] message. All others, NN, SNN, JT, TT are running [OK]. 1. Starting daemons, all seems OK: Starting Hadoop datanode: [ OK ] starting datanode, logging to /home/hadoop/logs/hadoop-root-datanode-ip-10-204-47-138.out Starting Hadoop namenode: [ OK ] starting namenode, logging to /home/hadoop/logs/hadoop-hdfs-namenode-ip-10-204-47-138.out Starting Hadoop secondarynamenode: [ OK ] starting secondarynamenode, logging to /home/hadoop/logs/hadoop-hdfs-secondarynamenode-ip-10-204-47-138.out 2. running service -status-all command and get: Hadoop datanode is not running [FAILED] Hadoop namenode is running [ OK ] Hadoop secondarynamenode is running [ OK ] 3. Here is log file on DN: 2012-10-25 15:33:37,554 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting DataNode STARTUP_MSG: host = ip-10-204-47-138.ec2.internal/10.204.47.138 STARTUP_MSG: args = [] STARTUP_MSG: version = 2.0.0-cdh4.1.1 STARTUP_MSG: classpath = /etc/ha.......... ............................... .............................. 2012-10-25 15:33:38,098 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoop/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 2012-10-25 15:33:41,589 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2012-10-25 15:33:42,125 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2012-10-25 15:33:42,125 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2012-10-25 15:33:42,204 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is ip-10-204-47-138.ec2.internal 2012-10-25 15:33:42,319 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:50010 2012-10-25 15:33:42,323 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s 2012-10-25 15:33:42,412 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2012-10-25 15:33:42,603 INFO org.apache.hadoop.http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode 2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2012-10-25 15:33:42,682 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 0.0.0.0:50075 2012-10-25 15:33:42,690 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false 2012-10-25 15:33:42,690 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075 2012-10-25 15:33:42,690 INFO org.mortbay.log: jetty-6.1.26.cloudera.2 2012-10-25 15:33:43,601 INFO org.mortbay.log: Started [email protected]:50075<mailto:[email protected]:50075> 2012-10-25 15:33:43,787 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 50020 2012-10-25 15:33:43,905 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:50020 2012-10-25 15:33:43,917 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null 2012-10-25 15:33:43,943 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default> 2012-10-25 15:33:43,950 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoop/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 2012-10-25 15:33:43,958 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (storage id unknown) service to localhost/127.0.0.1:8020 starting to offer service 2012-10-25 15:33:44,297 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2012-10-25 15:33:44,304 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting 2012-10-25 15:33:45,551 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:46,605 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:47,865 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:48,945 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:49,946 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:50,947 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:51,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:52,997 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:54,047 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS) 2012-10-25 15:33:57,579 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/hadoop/dfs/data/in_use.lock acquired by nodename [email protected]<mailto:[email protected]> 2012-10-25 15:33:59,279 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled 2012-10-25 15:33:59,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=587246899;bpid=BP-443540453-10.204.47.138-1351175525331;lv=-40;nsInfo=lv=-40;cid=CID-55808532-08a8-447c-b5e7-e609f6c546ee;nsid=587246899;c=0;bpid=BP-443540453-10.204.47.138-1351175525331 2012-10-25 15:33:59,457 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - /home/hadoop/dfs/data/current 2012-10-25 15:33:59,583 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean 2012-10-25 15:33:59,680 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1351195722679 with interval 21600000 2012-10-25 15:33:59,688 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-443540453-10.204.47.138-1351175525331 2012-10-25 15:34:00,339 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020 beginning handshake with NN 2012-10-25 15:34:00,485 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020 successfully registered with NN 2012-10-25 15:34:00,491 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode localhost/127.0.0.1:8020 using DELETEREPORT_INTERVAL of 300000 msec BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000 2012-10-25 15:34:01,106 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020 trying to claim ACTIVE state with txid=2697 2012-10-25 15:34:01,106 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020 2012-10-25 15:34:01,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 30 blocks took 8 msec to generate and 369 msecs for RPC and NN processing 2012-10-25 15:34:01,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@d2f41a5 2012-10-25 15:34:01,496 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-443540453-10.204.47.138-1351175525331. 2012-10-25 15:34:01,516 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-443540453-10.204.47.138-1351175525331 to blockPoolScannerMap, new size=1 2012-10-25 15:34:04,939 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-2780693186250942847_1214 2012-10-25 15:34:04,962 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_3001053775353836209_1049 2012-10-25 15:34:05,081 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-5830536650401518708_1776 2012-10-25 15:34:05,100 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-7796621539450888414_1161 2012-10-25 15:34:05,110 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_3113923085061591346_1318 2012-10-25 15:34:05,111 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-3360339171714568163_1477 2012-10-25 15:34:05,111 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-8153671645708824194_1779 2012-10-25 15:34:05,112 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-1321321546165212517_1002 2012-10-25 15:34:05,116 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-7769865739291371890_1479 Rgds, AK47 NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite. Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel
