Hello, We have a cluster with ~1.5TB of data on it. After the power is off unexpectedly, I restart the cluster again but the administrative webpage is not available. When I ran $hadoop fsck /, I got bellow error message:
had...@b1:~/hadoop/bin$ ./hadoop fsck / Exception in thread "main" java.io.IOException: Server returned HTTP response code: 503 for URL: http://0.0.0.0:50070/fsck?path=%2F at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1313) at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:123) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:159) had...@b1:~/hadoop/bin$ Checking the log of the namenode, bellow is what I see: 2010-08-04 06:21:57,624 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 192.168.1.5:50010 to delete blk_7399677479348617513_221053 2010-08-04 06:22:00,625 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* ask 192.168.1.1:50010 to delete blk_7399677479348617513_221053 2010-08-04 06:23:09,067 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.3 2010-08-04 06:28:09,073 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.3 2010-08-04 06:33:09,080 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.3 2010-08-04 06:38:09,086 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.3 2010-08-04 06:43:09,092 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.3 2010-08-04 06:48:09,098 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.1.3 And bellow is log of a datanode (other datanodes have quite similar log patten): 2010-08-04 06:21:18,070 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_7399677479348617513_221053 file /home/hadoop/hdfs/data/current/blk_7399677479348617513 2010-08-04 06:23:20,602 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-1342019480713368018_215616 2010-08-04 06:25:32,919 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-4748941997410465848_220293 2010-08-04 06:29:49,380 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_4296166818468293227_220300 2010-08-04 06:32:20,033 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_3798415444126765289_220122 2010-08-04 06:32:20,634 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-8132282518829193527_207589 Any help would be greatly appreciated, Thanks, ~Hieu
