[
https://issues.apache.org/jira/browse/MAPREDUCE-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027598#comment-13027598
]
Devaraj K commented on MAPREDUCE-2463:
--------------------------------------
As per MAPREDUCE-2351, if we give any arbitrary file system URI for
"mapred.job.tracker.history.completed.location" is working fine. But if we give
other than local file system URI for "mapreduce.jobtracker.jobhistory.location"
is failing.
{code:title=JobHistory.java|borderStyle=solid}
private void moveToDoneNow(Path fromPath, Path toPath) throws IOException {
//check if path exists, in case of retries it may not exist
if (logDirFs.exists(fromPath)) {
LOG.info("Moving " + fromPath.toString() + " to " +
toPath.toString());
doneDirFs.moveFromLocalFile(fromPath, toPath);
doneDirFs.setPermission(toPath,
new FsPermission(JobHistory.HISTORY_FILE_PERMISSION));
}
}
{code}
In the above code, doneDirFs.moveFromLocalFile(fromPath, toPath); is trying to
move from local file system even if we give the history location as hdfs
location and it is failing with the above exception.
> Job History files are not moving to done folder when job history location is
> hdfs location
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2463
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2463
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.23.0
> Reporter: Devaraj K
> Assignee: Devaraj K
>
> If "mapreduce.jobtracker.jobhistory.location" is configured as HDFS location
> then either during initialization of Job Tracker (while moving old job
> history files) or after completion of the job, history files are not moving
> to done and giving following exception.
> {code:xml}
> 2011-04-29 15:27:27,813 ERROR
> org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history
> file to DONE folder.
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected:
> file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
> at
> org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
