I used a directory that is local to every slave node: export HADOOP_LOG_DIR="/scratch/$USER/$PBS_JOBID/hadoop-$USER/log".
I did not change the " hadoop.job.history.user.location", I thought if I don't change this property, the job history is going to be stored in hdfs under output/_logs dir. Then after the job completes, I copied back the logs to the server. Thanks a lot, Boyu On Fri, Jun 7, 2013 at 2:32 PM, Shahab Yunus <[email protected]> wrote: > What value do you have for hadoop.log.dir property? > > > On Fri, Jun 7, 2013 at 5:20 PM, Boyu Zhang <[email protected]> wrote: > >> Hi Shahab, >> >> >> How old were they? >>> >> They are new, I did the copy automatically right after the job completed, >> in a script. >> >> I am assuming they were from the jobs run on the older version, right? >>> >> I run the job using the hadoop version 2.0.4 if this is what you mean. >> >> Or are you looking for new jobs's log that you are running after the >>> upgrade? >>> >> I am looking for new job's log, I did a fresh install of hadoop2.0.4, >> then run the job, then copied back the entire hdfs directory. >> >> >>> What about the local file systems? Are the logs there still? >>> >> Where should I find the logs (job logs, not daemon logs) in the local >> file systems? >> >> Thanks, >> Boyu >> >> >>> >>> Regards, >>> Shahab >>> >>> >>> On Fri, Jun 7, 2013 at 4:56 PM, Boyu Zhang <[email protected]>wrote: >>> >>>> Thanks Shahab, >>>> >>>> I saw the link, but it is not the case for me. I copied everything from >>>> hdfs ($HADOOP_HOME/bin/hdfs dfs -copyToLocal / $local_dir). But did not see >>>> the logs. >>>> >>>> Did it work for you? >>>> >>>> Thanks, >>>> Boyu >>>> >>>> >>>> On Fri, Jun 7, 2013 at 1:52 PM, Shahab Yunus <[email protected]>wrote: >>>> >>>>> See this; >>>>> >>>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201302.mbox/%[email protected]%3E >>>>> >>>>> Regards, >>>>> Shahab >>>>> >>>>> >>>>> On Fri, Jun 7, 2013 at 4:33 PM, Boyu Zhang <[email protected]>wrote: >>>>> >>>>>> Dear All, >>>>>> >>>>>> I recently moved from Hadoop0.20.2 to 2.0.4, and I am trying to find >>>>>> the old job history files (used to be in hdfs, output/_logs/history), it >>>>>> records detailed time information for every task attempts. >>>>>> >>>>>> But now it is not on hdfs anymore, I copied the entire "/" from hdfs >>>>>> to my local dir, but am not able to find this location. >>>>>> >>>>>> Could anyone give any advise on where are the files? >>>>>> >>>>>> Thanks, >>>>>> Boyu >>>>>> >>>>> >>>>> >>>> >>> >> >> >
