[
https://issues.apache.org/jira/browse/MAPREDUCE-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038547#comment-13038547
]
Devaraj K commented on MAPREDUCE-2427:
--------------------------------------
Hi Todd,
I don't see the code where it overwrites the history file.
{code:title=JobHistory.java|borderStyle=solid}
FileSystem fs = logDir.getFileSystem(conf);
if (!fs.exists(logDir)){
if (!fs.mkdirs(logDir, new FsPermission(HISTORY_DIR_PERMISSION))) {
throw new IOException("Mkdirs failed to create " + logDir.toString());
}
}
conf.set("hadoop.job.history.location", LOG_DIR);
{code}
At the time initialization, it checks for the history if it doesn't present it
will create history directory. It doesn't check existing history file is
directory or not. If it is not a directory it will fail in the below line while
creating a log file after job submission.
{code:title=JobHistory.java|borderStyle=solid}
out = fs.create(logFile,
new FsPermission(HISTORY_FILE_PERMISSION),
true,
defaultBufferSize,
fs.getDefaultReplication(),
jobHistoryBlockSize, null);
{code}
With the above line of code, if the history directory doesn't exist it will
create. If the existing history file is not a directory it fails saying
"java.io.IOException: Mkdirs failed to create".
> JT should ensure history directory is a directory
> -------------------------------------------------
>
> Key: MAPREDUCE-2427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2427
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.20.2
> Reporter: E. Sammer
> Assignee: Devaraj K
> Fix For: 0.20.4, 0.23.0
>
> Attachments: MAPREDUCE-2427.0.20.patch, MAPREDUCE-2427.patch
>
>
> If the JT history directory doesn't exist or isn't a directory retired job
> files are renamed to a file called 'history' and eventually start overwriting
> each other. The JT should ensure 'history' exists and is a directory before
> performing the move.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
