[
https://issues.apache.org/jira/browse/SPARK-13975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-13975.
----------------------------------
Resolution: Incomplete
> Cannot specify extra libs for executor from /extra-lib
> ------------------------------------------------------
>
> Key: SPARK-13975
> URL: https://issues.apache.org/jira/browse/SPARK-13975
> Project: Spark
> Issue Type: Improvement
> Components: Spark Submit
> Affects Versions: 1.6.1
> Reporter: Leonid Poliakov
> Priority: Minor
> Labels: bulk-closed
>
> If you build a framework on top of spark and want to bundle it with the
> spark, there is no easy way to add your framework libs to executor classpath.
> Let's say I want to add my custom libs to {{/extra-lib}} folder, ship the new
> bundle (with my libs in it) to nodes, run the bundle. I want executors on
> node to always automatically load my libs from {{/extra-lib}}, because that's
> how future developers would use framework out-of-the-box.
> The config doc says you can specify extraClasspath for the executor in
> {{spark-defaults.conf}}, which is good because custom config may be put in
> the bundle for the framework, but the syntax of the property is unclear.
> You can basically specify the value that will be appended to {{-cp}} for a
> executor Java process, so it follows the Java how-to-set-classpath rules, so
> basically you have two options here:
> 1. specify absolute path
> bq. spark.executor.extraClassPath /home/user/Apps/spark-bundled/extra-lib/*
> 2. specify relative path
> bq. spark.executor.extraClassPath ../../../extra-lib/*
> But none of these ways look good: absolute path won't work at all since you
> cannot know where users will put the bundle, relative path looks weird
> because executor will have it's work directory set to something like
> {{/work/app-20160316070310-0002/0}} and can also be broken if custom worker
> folder is configured.
> So, it's required to have a proper way to bundle custom libs and set executor
> classpath to load them up.
> *Expected*: you can specify {{spark.executor.extraClassPath}} relative to
> {{$SPARK_HOME}} using placeholders, e.g. with next syntax:
> bq. spark.executor.extraClassPath ${home}/extra-lib/*
> Code will resolve placeholders in properties with a proper path
> Executor will get absolute path in {{-cp}} this way
> *Actual*: you cannot specify extra libs for executor relative to
> {{$SPARK_HOME}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
