This looks like a log dir problem:
at
org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
looking through the source for JobLocalizer, it's trying to create a
folder under ${hadoop.log.dir}/userlogs. There's a similar question (i
assume it's yours) on StackOverflow:
http://stackoverflow.com/questions/9992566/hadoop-map-reduce-operation-is-failing-on-writing-output
Basically from your ps trace, the hadoop.log.dir is pointing to
/home/hadoopmachine/hadoop-1.0.1/libexec/../logs - check this folder
is writable by your 'hadoopmachine' user:
1000 4249 2.2 0.8 1181992 30176 ? Sl 12:09 0:00
/usr/lib/jvm/java-6-openjdk/bin/java -Dproc_tasktracker -Xmx1000m
-Dhadoop.log.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/../logs
-Dhadoop.log.file=hadoop-hadoopmachine-tasktracker-debian.log
-Dhadoop.home.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/..
-Dhadoop.id.str=hadoopmachine -Dhadoop.root.logger=INFO,DRFA
-Dhadoop.security.logger=INFO,NullAppender
-Djava.library.path=/home/hadoopmachine/hadoop-1.0.1/libexec/../lib/native/Linux-i386-32
-Dhadoop.policy.file=hadoop-policy.xml -classpath
On Tue, Apr 3, 2012 at 2:34 PM, Bas Hickendorff
<hickendorff...@gmail.com> wrote:
> ps -ef | grep hadoop shows that it is indeed "hadoopmachine" that is
> running hadoop.
>
> I su'ed into the user hadoopmachine (which is also the standard user I
> login with in debian), and I can access the hdfs that way as well.
>
> The free space should also not be a problem:
>
> hadoopmachine@debian:~$ df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 7.6G 4.0G 3.2G 56% /
> tmpfs 1.8G 0 1.8G 0% /lib/init/rw
> udev 1.8G 172K 1.8G 1% /dev
> tmpfs 1.8G 0 1.8G 0% /dev/shm
>
>
> I don't know if it is relevant, but it is on a virtual machine.
>
>
> Regards,
>
> Bas
>
>
> On Tue, Apr 3, 2012 at 8:17 PM, Harsh J <ha...@cloudera.com> wrote:
>> The permissions look alright if TT too is run by 'hadoopmachine'. Can
>> you also check if you have adequate space free, reported by df -h
>> /home/hadoopmachine?
>>
>> On Tue, Apr 3, 2012 at 10:28 PM, Bas Hickendorff
>> <hickendorff...@gmail.com> wrote:
>>> Thanks for your help!
>>> However, as far as I can see, the user has those rights.
>>>
>>> I have in mapred-ste.xml :
>>>
>>> <property>
>>> <name>mapred.local.dir</name>
>>> <value>/home/hadoopmachine/hadoop_data/mapred</value>
>>> <final>true</final>
>>> </property>
>>>
>>>
>>> and the directories look like this:
>>>
>>> hadoopmachine@debian:~$ cd /home/hadoopmachine/hadoop_data/mapred
>>> hadoopmachine@debian:~/hadoop_data/mapred$ ls -lah
>>> total 24K
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 .
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 ..
>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 taskTracker
>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 tt_log_tmp
>>> drwx------ 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 ttprivate
>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 08:28 userlogs
>>>
>>> hadoopmachine@debian:~/hadoop_data/mapred$ cd ..
>>> hadoopmachine@debian:~/hadoop_data$ ls -lah
>>> total 24K
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 .
>>> drwxr-xr-x 31 hadoopmachine hadoopmachine 4.0K Apr 3 12:08 ..
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 data
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 mapred
>>> drwxr-xr-x 5 hadoopmachine hadoopmachine 4.0K Apr 3 12:09 name
>>> drwxr-xr-x 4 hadoopmachine hadoopmachine 4.0K Apr 3 10:11 tmp
>>>
>>>
>>> As far as I can see (but my linux permissions knowledge might be
>>> failing) the user "hadoopmachine" has rights on these folders. I
>>> confirmed that that user is indeed the user that runs the TaskTracker.
>>>
>>> Are there any other things I could check?
>>>
>>>
>>> Regards,
>>>
>>> Bas
>>>
>>> On Tue, Apr 3, 2012 at 6:12 PM, Harsh J <ha...@cloudera.com> wrote:
>>>> Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
>>>> permissions set on them. Make sure they are owned by the user that
>>>> runs the TT service and have read/write permission at least for that
>>>> user.
>>>>
>>>> On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
>>>> <hickendorff...@gmail.com> wrote:
>>>>> Hello all,
>>>>>
>>>>> My map-reduce operation on Hadoop (running on Debian) is correctly
>>>>> starting and finding the input file. However, just after starting the
>>>>> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>>>> it does not state what file it cannot find, or where it is looking.
>>>>> Does someone now about what file error is? See below for the complete
>>>>> error.
>>>>>
>>>>> Since the java error is in the chmod() function (judging from the
>>>>> stack in the output), I assume it is a problem with the rights, but
>>>>> how do I know what rights to change if it gives me no path?
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Bas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The output of the job:
>>>>>
>>>>>
>>>>> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>>>> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>>>> /user/hadoopmachine/input /user/hadoopmachine/output
>>>>> Warning: $HADOOP_HOME is deprecated.
>>>>>
>>>>> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>> parsing the arguments. Applications should implement Tool for the
>>>>> same.
>>>>> ****hdfs://localhost:9000/user/hadoopmachine/input
>>>>> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to
>>>>> process : 1
>>>>> 12/04/03 08:05:08 INFO mapred.JobClient: Running job:
>>>>> job_201204030722_0004
>>>>> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>>>> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>>>> attempt_201204030722_0004_m_000002_0, Status : FAILED
>>>>> Error initializing attempt_201204030722_0004_m_000002_0:
>>>>> ENOENT: No such file or directory
>>>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>>>> at
>>>>> org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>>>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>>>>> at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>>>> at
>>>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>>>> at
>>>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>>>>> at
>>>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
>>>>> at
>>>>> org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>>>> at
>>>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>>>>> at
>>>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>>>>> at
>>>>> org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>>>>> at java.lang.Thread.run(Thread.java:636)
>>>>>
>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>
>>
>>
>> --
>> Harsh J
