Thanks Marcelo. Do you think it would be useful to make
spark.executor.extraClassPath be made to pick up some environment variable
that can be set from spark-env.sh? Here is a example.
spark-env.sh
------------------
executor_extra_cp = get_hbase_jars_for_cp
export executor_extra_cp
spark-defaults.conf
---------------------
spark.executor.extraClassPath = ${executor_extra_cp}
This will let us add logic inside get_hbase_jars_for_cp function to pick
the right version hbase jars. There could be multiple versions installed on
the node.
--
Kannan
On Thu, Feb 26, 2015 at 6:08 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
> On Thu, Feb 26, 2015 at 5:12 PM, Kannan Rajah <kra...@maprtech.com> wrote:
> > Also, I would like to know if there is a localization overhead when we
> use
> > spark.executor.extraClassPath. Again, in the case of hbase, these jars
> would
> > be typically available on all nodes. So there is no need to localize them
> > from the node where job was submitted. I am wondering if we use the
> > SPARK_CLASSPATH approach, then it would not do localization. That would
> be
> > an added benefit.
> > Please clarify.
>
> spark.executor.extraClassPath doesn't localize anything. It just
> prepends those classpath entries to the usual classpath used to launch
> the executor. There's no copying of files or anything, so they're
> expected to exist on the nodes.
>
> It's basically exactly the same as SPARK_CLASSPATH, but broken down to
> two options (one for the executors, and one for the driver).
>
> --
> Marcelo
>
