[
https://issues.apache.org/jira/browse/MAPREDUCE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070365#comment-15070365
]
Sangjin Lee commented on MAPREDUCE-6588:
----------------------------------------
One interesting observation I made while trying to use the cross-platform
macros for the MR AM as a test:
{noformat}
PATH={{PATH}};{{HADOOP_COMMON_HOME}}\bin
{noformat}
The node manager sets the following line in the batch script that launches the
AM:
{noformat}
@set PATH=%PATH%;%HADOOP_COMMON_HOME%\bin
{noformat}
However, on the AM process, the PATH environment variable is as follows:
{noformat}
PATH=%PATH%;%HADOOP_COMMON_HOME%\bin;C:\hadoop\hadoop-mapreduce-project\hadoop-mapreduce-client\hadoop-mapreduce-client-jobclient/../../../hadoop-common-project/hadoop-common/target\bin
{noformat}
While the substitution for HADOOP_COMMON_HOME is made, PATH is not. And also
the original macro is preserved instead of being replaced. Maybe this is
because I'm not familiar with Windows batch scripting, but this causes the
existing path value to be dropped.
> default values for PATH for tasks are not quite correct
> -------------------------------------------------------
>
> Key: MAPREDUCE-6588
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6588
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
> Affects Versions: 2.6.0
> Reporter: Sangjin Lee
>
> There are several issues with the way PATH are handled for MR tasks and AMs
> on Windows.
> - it does not get the PWD as part of the path, unlike the non-Windows case
> (see MAPREDUCE-6021)
> - the MR AM doesn't get the Hadoop's bin directory in the path explicitly
> through mapred configuration; rather, it implicitly inherits it from the node
> manager (see MAPREDUCE-6577)
> - the default value for the MR task PATH uses "%...%" macros which may be
> expanded on the client side prematurely; this is probably OK for tasks (as
> it's done by the AM), but not OK for AMs; the late-binding cross-platform
> macros should be used
> I took a stab at addressing these as part of MAPREDUCE-6577, but I realized
> that it would be a bigger issue than the fix for MAPREDUCE-6577 itself. Thus,
> I'm filing a separate JIRA to address the Windows side of things.
> On a related note, there are quite a few brittle tests on testing these
> (TestMiniMRChildTask, TestMiniMRClientCluster, etc.) which are built in such
> a way that the tests pass against the current behavior but don't provide a
> lot of robust test value. We should also update those tests to make them more
> robust.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
