[ 
https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077729#comment-14077729
 ] 

Jason Lowe commented on YARN-2369:
----------------------------------

The code in question is in org.apache.hadoop.yarn.util.Apps#addToEnvironment:

{code}
  public static void addToEnvironment(
      Map<String, String> environment,
      String variable, String value, String classPathSeparator) {
    String val = environment.get(variable);
    if (val == null) {
      val = value;
    } else {
      val = val + classPathSeparator + value;
    }
    environment.put(StringInterner.weakIntern(variable), 
        StringInterner.weakIntern(val));
  }
{code}

This has very surprising results for any variable that isn't path-like.  For 
example, we ran across a MapReduce job that had something like this in its 
environment settings:

yarn.app.mapreduce.am.env='JAVA_HOME=/inst/jdk,JAVA_HOME=/inst/jdk'

Rather than ending up with JAVA_HOME=/inst/jdk as one would expect, JAVA_HOME 
instead was set to /inst/jdk:/inst/jdk which completely broke the job.

It seems to me that we should either use a whitelist of variables that support 
appending or never append settings.  For the latter case if users desire values 
to be appended then they can ask for it explicitly in their variable settings, 
like one of these forms depending upon whether they want client-side 
environment variable expansion or container-side environment variable expansion:
{noformat}
PATH='$PATH:/my/extra/path'
PATH='{{PATH}}:/my/extra/path'
{noformat}

> Environment variable handling assumes values should be appended
> ---------------------------------------------------------------
>
>                 Key: YARN-2369
>                 URL: https://issues.apache.org/jira/browse/YARN-2369
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Jason Lowe
>
> When processing environment variables for a container context the code 
> assumes that the value should be appended to any pre-existing value in the 
> environment.  This may be desired behavior for handling path-like environment 
> variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a 
> non-intuitive and harmful way to handle any variable that does not have 
> path-like semantics.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to