[
https://issues.apache.org/jira/browse/MAPREDUCE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096886#comment-14096886
]
Benjamin Zhitomirsky commented on MAPREDUCE-6032:
-------------------------------------------------
{quote}1. Doesn't need "JobHistoryUtils: ", the logger will take care of the
source where the log record is generated.{quote} - Fixed
{quote}2. Maybe you want to make the logic here clearer.{quote} - While your
version is more readable it will fail on sameFS line with NullPointerException.
This is because in the 'if' all conditions are ORed and if first condition is
false then the next won't be calculated. That's why it is better to put
everything in the 'if' in this case.
{quote}2. ...And path.toUri().getAuthority() != null ||
path.toUri().getScheme()!= null -> &&?{quote} OR is more proper here because if
path is partially qualified (e.g. there is a schema without autority) then we
don't want to mess with it and just return it as-is. Consider following example:
{code}new Path("s3:///dir/subdir").makeQualified(new URI("hdfs://server:8020"),
new Path("/dir"){code}
will return the following nonsense
{code}s3://server:8020/dir/subdir{code}
{quote}3. Is it possible to add a test case in TestJobHistoryEventHandler to
verify the JobHistoryHandler will write to the default FS as well?{quote} - Done
{quote}4. You may wan to fix the indents (for those sentences in multiple
lines).{quote} - You was not specific, therefore please check that I fixed
everything you meant.
{quote}5. Unlike makeQualified, the following method will work when
stagingDirPath is on a different FS than the configure one, right?{quote} -
Right
I will upload the updated patch soon.
> Unable to check mapreduce job status if submitted using a non-default namenode
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6032
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6032
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 2.0.5-alpha, 2.1.1-beta, 2.0.6-alpha, 2.2.0, 2.3.0,
> 2.2.1, 2.4.1
> Environment: Any
> Reporter: Benjamin Zhitomirsky
> Assignee: Benjamin Zhitomirsky
> Fix For: trunk
>
> Attachments: MAPREDUCE-6032.patch
>
> Original Estimate: 24h
> Time Spent: 24h
> Remaining Estimate: 0h
>
> When MRv2 job container runs in a context of non-default file system
> JobHistoryUtils.java obtains mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir as a non-qualified paths (e.g.
> /mapred/history). This path is considered to belong to the current
> container's context. As result the application history is being written to
> another file system and job history server is unable to pick it up, because
> it expects it to be found on the default file system. Currently providing
> fully qualified path to those parameters is not supported as well, because of
> a bug in JobHistoryEventHandler.
> After this fix two scenarios will be supported:
> - mapreduce.jobhistory.done-dir and
> mapreduce.jobhistory.intermediate-done-dir (and the staging directory BTW)
> will support a fully qualified path
> - If a non-qualified path is configured then it will always be defaulted to
> the default file system (core-site.xml). That's how consistency of history
> location will be archived
> Implementation notes:
> - FileSystem#makeQualified throws exception if specified path belongs to
> another file system. However FileContext#makeQualified work properly in this
> case, and this is the meaning of the fix in JobHistoryEventHandler. I was not
> ready to change behavior FileSystem#makeQualified because much more thought
> is required. I afraid that many users expect such behavior, and fixing it
> would break their code.
> - The fix in JobHistoryUtils detects non-default namenode configuration only
> if it comes from some "real" configuration: core-default.xml is ignored. This
> is done primary as a kind of test hook, because otherwise setting
> fs.defaultFS value during test executions would be always recognized by
> JobHistoryUtils as a non-default namenode against 'file:///' specified in
> core-default.xml.
> (Remark. Note that makeQualified doesn't behave properly with file:///
> filesystem, for example:
> new Path("file:///dir/subdir").makeQualified(new URI("hdfs://server:8020"),
> new Path("/dir"))
> Returns: "file://server:8020/dir/subdir" which doesn't make sense.
> However I don't believe it worth fixing, since nobody really case about local
> file system besides tests. My fix just ensures that all tests run smoothly by
> ignoring core-default.xml file system in the logic.)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
