[
https://issues.apache.org/jira/browse/MAPREDUCE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092801#comment-14092801
]
Hadoop QA commented on MAPREDUCE-6032:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660984/MAPREDUCE-6032.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4798//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4798//console
This message is automatically generated.
> Unable to check mapreduce job status if submitted using a non-default namenode
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6032
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6032
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.5-alpha, 2.1.1-beta, 2.0.6-alpha, 2.2.0, 2.3.0,
> 2.2.1, 2.4.1
> Environment: Any
> Reporter: Benjamin Zhitomirsky
> Assignee: Benjamin Zhitomirsky
> Fix For: trunk
>
> Attachments: MAPREDUCE-6032.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When MRv2 job container runs in a context of non-default file system
> JobHistoryUtils.java obtains mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir as a non-qualified paths (e.g.
> /mapred/history). This path is considered to belong to the current
> container's context. As result the application history is being written to
> another file system and job history server is unable to pick it up, because
> it expects it to be found on the default file system. Currently providing
> fully qualified path to those parameters is not supported as well, because of
> a bug in JobHistoryEventHandler.
> After this fix two scenarios will be supported:
> - mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir (and the staging directory BTW)
> will support a fully qualified path
> - If a non-qualified path is configured then it will always be defaulted to
> the default file system (core-site.xml). That's how consistency of history
> location will be archived
> Implementation notes:
> - FileSystem#makeQualified throws exception if specified path belongs to
> another file system. However FileContext#makeQualified work properly in this
> case, and this is the meaning of the fix in JobHistoryEventHandler. I was not
> ready to change behavior FileSystem#makeQualified because much more thought
> is required. I afraid that many users expect such behavior, and fixing it
> would break their code.
> - The fix in JobHistoryUtils detects non-default namenode configuration only
> if it comes from some "real" configuration: core-default.xml is ignored. This
> is done primary as a kind of test hook, because otherwise setting
> fs.defaultFS value during test executions would be always recognized by
> JobHistoryUtils as a non-default namenode against 'file:///' specified in
> core-default.xml.
> (Remark. Note that makeQualified doesn't behave properly with file:///
> filesystem, for example:
> new Path("file:///dir/subdir").makeQualified(new URI("hdfs://server:8020"),
> new Path("/dir"))
> Returns: "file://server:8020/dir/subdir" which doesn't make sense.
> However I don't believe it worth fixing, since nobody really case about local
> file system besides tests. My fix just ensures that all tests run smoothly by
> ignoring core-default.xml file system in the logic.)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
