[ https://issues.apache.org/jira/browse/SPARK-13975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199662#comment-15199662 ]
Sean Owen commented on SPARK-13975: ----------------------------------- You would generally control the environment Spark runs in, and would use this to put files in a known uniform place on the classpath. Dependent libraries should be bundled with your app though if you need to control their distribution. > Cannot specify extra libs for executor from /extra-lib > ------------------------------------------------------ > > Key: SPARK-13975 > URL: https://issues.apache.org/jira/browse/SPARK-13975 > Project: Spark > Issue Type: Bug > Components: Spark Submit > Affects Versions: 1.6.1 > Reporter: Leonid Poliakov > > If you build a framework on top of spark and want to bundle it with the > spark, there is no easy way to add your framework libs to executor classpath. > Let's say I want to add my custom libs to {{/extra-lib}} folder, ship the new > bundle (with my libs in it) to nodes, run the bundle. I want executors on > node to always automatically load my libs from {{/extra-lib}}, because that's > how future developers would use framework out-of-the-box. > The config doc says you can specify extraClasspath for the executor in > {{spark-defaults.conf}}, which is good because custom config may be put in > the bundle for the framework, but the syntax of the property is unclear. > You can basically specify the value that will be appended to {{-cp}} for a > executor Java process, so it follows the Java how-to-set-classpath rules, so > basically you have two options here: > 1. specify absolute path > bq. spark.executor.extraClassPath /home/user/Apps/spark-bundled/extra-lib/* > 2. specify relative path > bq. spark.executor.extraClassPath ../../../extra-lib/* > But none of these ways look good: absolute path won't work at all since you > cannot know where users will put the bundle, relative path looks weird > because executor will have it's work directory set to something like > {{/work/app-20160316070310-0002/0}} and can also be broken if custom worker > folder is configured. > So, it's required to have a proper way to bundle custom libs and set executor > classpath to load them up. > *Expected*: you can specify {{spark.executor.extraClassPath}} relative to > {{$SPARK_HOME}} using placeholders, e.g. with next syntax: > bq. spark.executor.extraClassPath ${home}/extra-lib/* > Code will resolve placeholders in properties with a proper path > Executor will get absolute path in {{-cp}} this way > *Actual*: you cannot specify extra libs for executor relative to > {{$SPARK_HOME}} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org