[ 
https://issues.apache.org/jira/browse/YARN-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160255#comment-14160255
 ] 

Doug Haigh commented on YARN-2625:
----------------------------------

Yes, I agree the classpath should be set by the client, but the REST client 
should not have to know the default classpath just as the Java client does not 
need to know it. Just as the REST API resolves {{<CPS>}} to either {{:}} or 
{{;}} based on the underlying operating system, the REST API could look for 
{{<DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH>}} and resolve it to 
{{YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH}}.


As for environment variables being resolved, when running a Java client against 
a CDH 5.0.0 cluster, I am able to set the environment to 

{{./*:$HADOOP_CLIENT_CONF_DIR:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*}}

and it works - the environment variables are resolved. Maybe it is the way CDH 
is setting things up, but the path is not fully resolved when the client 
specifies it.

> Problems with CLASSPATH in Job Submission REST API
> --------------------------------------------------
>
>                 Key: YARN-2625
>                 URL: https://issues.apache.org/jira/browse/YARN-2625
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: api
>    Affects Versions: 2.5.1
>            Reporter: Doug Haigh
>
> There are a couple of issues I have found specifying the CLASSPATH 
> environment variable using the REST API.
> 1) In the Java client, the CLASSPATH environment is usually made up of either 
> the value of the yarn.application.classpath in yarn-site.xml value or the 
> default YARN classpath value as defined by 
> YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH. REST API 
> consumers have no method of telling the resource manager to use the default 
> unless they hardcode the default value themselves. If the default ever 
> changes, the code would need to change. 
> 2) If any environment variables are used in the CLASSPATH environment 'value' 
> field, they are evaluated when the values are NULL resulting in bad values in 
> the CLASSPATH. For example, if I had hardcoded the CLASSPATH value to the 
> default of "$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, 
> $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, 
> $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, 
> $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, 
> $HADOOP_YARN_HOME/share/hadoop/yarn/*, 
> $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*" the classpath passed to the 
> application master is 
> ":/share/hadoop/common/*:/share/hadoop/common/lib/*:/share/hadoop/hdfs/*:/share/hadoop/hdfs/lib/*:/share/hadoop/yarn/*:/share/hadoop/yarn/lib/*"
> These two problems require REST API consumers to always have the fully 
> resolved path defined in the yarn.application.classpath value. If the 
> property is missing or contains environment varaibles, the application 
> created by the REST API will fail due to the CLASSPATH being incorrect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to