[ 
https://issues.apache.org/jira/browse/YARN-2625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257618#comment-14257618
 ] 

Vinod Kumar Vavilapalli commented on YARN-2625:
-----------------------------------------------

bq.  If the default ever changes, the code would need to change.
The way the layering works, YARN doesn't know whether the app is JVM based or 
not (though predominant apps that exist today are all JVM based). Given this, 
we chose to introduce this yarn.application.classpath as a _simplifying_ 
configuration property that apps can use. So far, it didn't have any semantic 
meaning to the platform and it is important to keep it so to keep it so. I 
think the right solution is for us to introduce some sort of a JAVA universe as 
a API, we can auto-expand it the way you are asking only if the app 
specifically says it is a JAVA universe.

bq. For example, if I had hardcoded the CLASSPATH value to the default of 
"$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/, 
$HADOOP_COMMON_HOME/share/hadoop/common/lib/, 
$HADOOP_HDFS_HOME/share/hadoop/hdfs/, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/, 
$HADOOP_YARN_HOME/share/hadoop/yarn/, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/" 
the classpath passed to the application master is 
":/share/hadoop/common/:/share/hadoop/common/lib/:/share/hadoop/hdfs/:/share/hadoop/hdfs/lib/:/share/hadoop/yarn/:/share/hadoop/yarn/lib/"
This is a bug. NodeManager by default exports these environment variables and 
so the expansion should happen automatically.

Are you using MR jobs? If so, once your clusters start moving to using MR 
tarballs through distributed-cache scheme (MAPREDUCE-4421), the classpath 
dictated by mapreduce.application.classpath will simply be a reflection of the 
directory hierarchy in the tarball and is completely decoupled from the 
cluster-install or the environment variables.

> Problems with CLASSPATH in Job Submission REST API
> --------------------------------------------------
>
>                 Key: YARN-2625
>                 URL: https://issues.apache.org/jira/browse/YARN-2625
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: api
>    Affects Versions: 2.5.1
>            Reporter: Doug Haigh
>
> There are a couple of issues I have found specifying the CLASSPATH 
> environment variable using the REST API.
> 1) In the Java client, the CLASSPATH environment is usually made up of either 
> the value of the yarn.application.classpath in yarn-site.xml value or the 
> default YARN classpath value as defined by 
> YarnConfiguration.DEFAULT_YARN_CROSS_PLATFORM_APPLICATION_CLASSPATH. REST API 
> consumers have no method of telling the resource manager to use the default 
> unless they hardcode the default value themselves. If the default ever 
> changes, the code would need to change. 
> 2) If any environment variables are used in the CLASSPATH environment 'value' 
> field, they are evaluated when the values are NULL resulting in bad values in 
> the CLASSPATH. For example, if I had hardcoded the CLASSPATH value to the 
> default of "$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, 
> $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, 
> $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, 
> $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, 
> $HADOOP_YARN_HOME/share/hadoop/yarn/*, 
> $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*" the classpath passed to the 
> application master is 
> ":/share/hadoop/common/*:/share/hadoop/common/lib/*:/share/hadoop/hdfs/*:/share/hadoop/hdfs/lib/*:/share/hadoop/yarn/*:/share/hadoop/yarn/lib/*"
> These two problems require REST API consumers to always have the fully 
> resolved path defined in the yarn.application.classpath value. If the 
> property is missing or contains environment varaibles, the application 
> created by the REST API will fail due to the CLASSPATH being incorrect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to