Hello,
Yes, that question on stackoverflow is mine as well. I first posted
there, but then realized that in this mailgroup there would probably
be more Hadoop knowledge.
And you where right! For some reason the userlog folder in the log
folder was owned by root instead of hadoopmachine. I have no idea how
that happened, but I changed it and now it is running without errors.
Everybody thanks a lot for you time and help, much appreciated!
Chris: I will accept your answer on stackoverflow.
Regards,
Bas
On Tue, Apr 3, 2012 at 8:57 PM, Chris White <chriswhite...@gmail.com> wrote:
> This looks like a log dir problem:
>
> at
> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>
> looking through the source for JobLocalizer, it's trying to create a
> folder under ${hadoop.log.dir}/userlogs. There's a similar question (i
> assume it's yours) on StackOverflow:
>
> http://stackoverflow.com/questions/9992566/hadoop-map-reduce-operation-is-failing-on-writing-output
>
> Basically from your ps trace, the hadoop.log.dir is pointing to
> /home/hadoopmachine/hadoop-1.0.1/libexec/../logs - check this folder
> is writable by your 'hadoopmachine' user:
>
> 1000 4249 2.2 0.8 1181992 30176 ? Sl 12:09 0:00
> /usr/lib/jvm/java-6-openjdk/bin/java -Dproc_tasktracker -Xmx1000m
> -Dhadoop.log.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/../logs
> -Dhadoop.log.file=hadoop-hadoopmachine-tasktracker-debian.log
> -Dhadoop.home.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/..
> -Dhadoop.id.str=hadoopmachine -Dhadoop.root.logger=INFO,DRFA
> -Dhadoop.security.logger=INFO,NullAppender
> -Djava.library.path=/home/hadoopmachine/hadoop-1.0.1/libexec/../lib/native/Linux-i386-32
> -Dhadoop.policy.file=hadoop-policy.xml -classpath
>
>
>
> On Tue, Apr 3, 2012 at 2:34 PM, Bas Hickendorff
> <hickendorff...@gmail.com> wrote:
>> ps -ef | grep hadoop shows that it is indeed "hadoopmachine" that is
>> running hadoop.
>>
>> I su'ed into the user hadoopmachine (which is also the standard user I
>> login with in debian), and I can access the hdfs that way as well.
>>
>> The free space should also not be a problem:
>>
>> hadoopmachine@debian:~$ df -h
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda1 7.6G 4.0G 3.2G 56% /
>> tmpfs 1.8G 0 1.8G 0% /lib/init/rw
>> udev 1.8G 172K 1.8G 1% /dev
>> tmpfs 1.8G 0 1.8G 0% /dev/shm
>>
>>
>> I don't know if it is relevant, but it is on a virtual machine.
>>
>>
>> Regards,
>>
>> Bas
>>
>>
>> On Tue, Apr 3, 2012 at 8:17 PM, Harsh J <ha...@cloudera.com> wrote:
>>> The permissions look alright if TT too is run by 'hadoopmachine'. Can
>>> you also check if you have adequate space free, reported by df -h
>>> /home/hadoopmachine?
>>>
>>> On Tue, Apr 3, 2012 at 10:28 PM, Bas Hickendorff
>>> <hickendorff...@gmail.com> wrote:
>>>> Thanks for your help!
>>>> However, as far as I can see, the user has those rights.
>>>>
>>>> I have in mapred-ste.xml :
>>>>
>>>> <property>
>>>> <name>mapred.local.dir</name>
>>>> <value>/home/hadoopmachine/hadoop_data/mapred</value>
>>>> <final>true</final>
>>>> </property>
>>>>
>>>>
>>>> and the directories look like this:
>>>>
>>>> hadoopmachine@debian:~$ cd /home/hadoopmachine/hadoop_data/mapred
>>>> hadoopmachine@debian:~/hadoop_data/mapred$ ls -lah
>>>> total 24K
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 .
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 ..
>>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 taskTracker
>>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 tt_log_tmp
>>>> drwx------ 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 ttprivate
>>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 08:28 userlogs
>>>>
>>>> hadoopmachine@debian:~/hadoop_data/mapred$ cd ..
>>>> hadoopmachine@debian:~/hadoop_data$ ls -lah
>>>> total 24K
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 .
>>>> drwxr-xr-x 31 hadoopmachine hadoopmachine 4.0K Apr 3 12:08 ..
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 data
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 mapred
>>>> drwxr-xr-x 5 hadoopmachine hadoopmachine 4.0K Apr 3 12:09 name
>>>> drwxr-xr-x 4 hadoopmachine hadoopmachine 4.0K Apr 3 10:11 tmp
>>>>
>>>>
>>>> As far as I can see (but my linux permissions knowledge might be
>>>> failing) the user "hadoopmachine" has rights on these folders. I
>>>> confirmed that that user is indeed the user that runs the TaskTracker.
>>>>
>>>> Are there any other things I could check?
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Bas
>>>>
>>>> On Tue, Apr 3, 2012 at 6:12 PM, Harsh J <ha...@cloudera.com> wrote:
>>>>> Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
>>>>> permissions set on them. Make sure they are owned by the user that
>>>>> runs the TT service and have read/write permission at least for that
>>>>> user.
>>>>>
>>>>> On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
>>>>> <hickendorff...@gmail.com> wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> My map-reduce operation on Hadoop (running on Debian) is correctly
>>>>>> starting and finding the input file. However, just after starting the
>>>>>> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>>>>> it does not state what file it cannot find, or where it is looking.
>>>>>> Does someone now about what file error is? See below for the complete
>>>>>> error.
>>>>>>
>>>>>> Since the java error is in the chmod() function (judging from the
>>>>>> stack in the output), I assume it is a problem with the rights, but
>>>>>> how do I know what rights to change if it gives me no path?
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Bas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The output of the job:
>>>>>>
>>>>>>
>>>>>> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>>>>> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>>>>> /user/hadoopmachine/input /user/hadoopmachine/output
>>>>>> Warning: $HADOOP_HOME is deprecated.
>>>>>>
>>>>>> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>>> parsing the arguments. Applications should implement Tool for the
>>>>>> same.
>>>>>> ****hdfs://localhost:9000/user/hadoopmachine/input
>>>>>> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to
>>>>>> process : 1
>>>>>> 12/04/03 08:05:08 INFO mapred.JobClient: Running job:
>>>>>> job_201204030722_0004
>>>>>> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>>>>> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>>>>> attempt_201204030722_0004_m_000002_0, Status : FAILED
>>>>>> Error initializing attempt_201204030722_0004_m_000002_0:
>>>>>> ENOENT: No such file or directory
>>>>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>>>>> at
>>>>>> org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>>>>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>>>>>> at
>>>>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>>>>> at
>>>>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>>>>>> at java.lang.Thread.run(Thread.java:636)
>>>>>>
>>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
>>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Harsh J
