Daryn Sharp created MAPREDUCE-4877:
--------------------------------------
Summary: AM doesn't properly support multiple NNs
Key: MAPREDUCE-4877
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4877
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, job submission
Affects Versions: 2.0.0-alpha, 0.23.0, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Yarn/MR clusters assume there's a 1-to-1 correspondence between itself and a
NN. Certain internal paths like the staging dir, job history,
intermediate/intermediate-done dirs are resolved relative to the defaultFS.
The JT used the host's conf which ensured the correct/expected NN. However the
AM uses the user's job conf, which means the user's defined defaultFS can cause
the job to use incorrect paths.
Typically the output path's NN is also the yarn cluster's NN. However problems
occur when a yarn cluster is servicing multiple NN's (ex. federated clusters).
The JHS is assuming the AM will write to NN1, whereas the user's job conf may
be using a defaultFS of NN2 or NN3 which influences where the AM writes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
