I have a wrecked datanode which is giving me hard time restarting. It keeps complaining of Datanode dead, pid file exists. I already tried deleting the files but seems like the files are corrupted and don't allow me delete.
____________________________________________________________________ Here is the log: ____________________________________________________________________ /************************************************************ STARTUP_MSG: Starting DataNode STARTUP_MSG: host = hadoop20/192.168.1.190 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.2-cdh3u0 STARTUP_MSG: build = -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14; compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011 ************************************************************/ 2011-06-06 09:11:01,232 INFO org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing. 2011-06-06 09:11:01,369 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.util.Shell$ExitCodeException: du: cannot access `/part/data/tmp/distcache/642114211252449475_2038269146_79 9583695/hmaster/user/mapred/input-dir': No such file or directory du: cannot read directory `/part/data/tmp/mapred/jobcache/job_201105261845_0005': Permission denied _________________________________ Here is the file I can't delete _________________________________ [root@hadoop20 distcache]# pwd /part/data/tmp/distcache [root@hadoop20 distcache]# ls -la total 0 drwxr-xr-x 3 mapred mapred 52 May 26 21:36 . drwxr-xr-x 4 mapred mapred 35 May 26 21:02 .. drwxr-xr-x 3 mapred mapred 20 May 26 21:17 642114211252449475_2038269146_799583695 [root@hadoop20 distcache]# cd * [root@hadoop20 642114211252449475_2038269146_799583695]# ls -la total 0 drwxr-xr-x 3 mapred mapred 20 May 26 21:17 . drwxr-xr-x 3 mapred mapred 52 May 26 21:36 .. drwxr-xr-x 3 mapred mapred 17 May 26 21:17 hmaster [root@hadoop20 642114211252449475_2038269146_799583695]# cd h* [root@hadoop20 hmaster]# ls user [root@hadoop20 hmaster]# cd * [root@hadoop20 user]# ls -la total 0 drwxr-xr-x 3 mapred mapred 19 May 26 21:17 . drwxr-xr-x 3 mapred mapred 17 May 26 21:17 .. drwxr-xr-x 3 mapred mapred 22 May 26 21:17 mapred [root@hadoop20 user]# cd m* [root@hadoop20 mapred]# ls -la total 0 drwxr-xr-x 3 mapred mapred 22 May 26 21:17 . drwxr-xr-x 3 mapred mapred 19 May 26 21:17 .. ?--------- ? ? ? ? ? input-dir [root@hadoop20 mapred]# rm input-dir rm: cannot lstat `input-dir': No such file or directory [root@hadoop20 mapred]# touch * [root@hadoop20 mapred]# ls input-dir input-dir [root@hadoop20 mapred]# rm * rm: remove regular empty file `input-dir'? y rm: cannot lstat `input-dir': No such file or directory [root@hadoop20 mapred]# ls -la total 0 drwxr-xr-x 3 mapred mapred 22 Jun 6 12:45 . drwxr-xr-x 3 mapred mapred 19 May 26 21:17 .. ?--------- ? ? ? ? ? input-dir [root@hadoop20 mapred]#
