[
https://issues.apache.org/jira/browse/MAPREDUCE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12893929#action_12893929
]
Ramya R commented on MAPREDUCE-1986:
------------------------------------
I have the following setup on my cluster:
<property>
<name>hadoop.job.history.location</name>
<value>/mapred/history1</value>
<description> If job tracker is static the history files are stored in this
single well known place. If No
value is set here, by default,
it is in the local file system at ${hadoop.log.dir}/history.
</description></property>
<property>
<name>mapred.job.tracker.history.completed.location</name>
<value>/mapred/history2/done</value>
<description> The completed job history files are stored at this single well
known location. If nothing is
specified, the files are stored at
${hadoop.job.history.location}/done.
</description>
</property>
i.e. both hadoop.job.history.location and
mapred.job.tracker.history.completed.location are configured to HDFS path.
The jobtracker successfully comes up without any error. However when the job
completes and it retires, I get the following exception:
{noformat}
ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
canonical subfolder.
java.io.FileNotFoundException: File
/mapred/history1/job_201007290838_0001_1280392750589_rramya_Sleep+job does not
exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1118)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1094)
at
org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1084)
at
org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager$1.run(JobHistory.java:300)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
However, /mapred/history1 is created and the history files are very much
present on HDFS.
Thus the history files remain forever in hadoop.job.history.location and are
never moved to mapred.job.tracker.history.completed.location. The jobhistory is
never viewable thereafter.
> History files fail to move to DONE folder when hadoop.job.history.location is
> configured to a HDFS path
> -------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1986
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: Ramya R
>
> When hadoop.job.history.location is configured to a HDFS path and when the
> job retires, the history files are never moved to
> mapred.job.tracker.history.completed.location and they remain forever in
> hadoop.job.history.location . The jobhistory is never viewable thereafter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
