[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150530#comment-16150530 ]
Thomas Graves edited comment on SPARK-21888 at 9/1/17 1:37 PM: --------------------------------------------------------------- Putting things into SPARK_CONF_DIR will work, the question is more about convenience for users. In hosted/multitenant environments there is probably a generic SPARK_CONF_DIR shared by everyone (at least this is how our env works), for the user to add hbase-site.xml they would have to copy, add files and then export SPARK_CONF_DIR. If that user continues to use the copied version they might miss changes to the cluster version, etc. Previously they didn't have to do this, they just had to set SPARK_CLASSPATH, of course even that doesn't always work if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it. So the question is more of what we think about this for convenience for users. Personally I think it would be nice to have a config that would allow users to set an extra classpath on the client side without having to modify the SPARK_CONF_DIR. thoughts from others? I think we can move this to an improvement jira, if other people here don't agree or see the usefulness then we can just close. was (Author: tgraves): Putting things into SPARK_CONF_DIR will work, the question is more about convenience for users. In hosted/multitenant environments there is probably a generic SPARK_CONF_DIR shared by everyone (at least this is how our env works), for the user to add hbase-site.xml they would have to copy, add files and then export SPARK_CONF_DIR. If that user continues to use the copied version they might miss changes to the cluster version, etc. Previously they didn't have to do this, they just had to set SPARK_CLASSPATH, of course even that doesn't always work if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it. So the question is more of what we think about this for convenience for users. Personally I think it would be nice to have a config that would allow users to set an extra classpath on the client side without having to modify the SPARK_CONF_DIR? I think we can move this to an improvement jira, if other people here don't agree or see the usefulness then we can just close. > Cannot add stuff to Client Classpath for Yarn Cluster Mode > ---------------------------------------------------------- > > Key: SPARK-21888 > URL: https://issues.apache.org/jira/browse/SPARK-21888 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.2.0 > Reporter: Parth Gandhi > Priority: Minor > > While running Spark on Yarn in cluster mode, currently there is no way to add > any config files, jars etc. to Client classpath. An example for this is that > suppose you want to run an application that uses hbase. Then, unless and > until we do not copy the necessary config files required by hbase to Spark > Config folder, we cannot specify or set their exact locations in classpath on > Client end which we could do so earlier by setting the environment variable > "SPARK_CLASSPATH". -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org