On Thu, Feb 26, 2015 at 5:12 PM, Kannan Rajah <kra...@maprtech.com> wrote: > Also, I would like to know if there is a localization overhead when we use > spark.executor.extraClassPath. Again, in the case of hbase, these jars would > be typically available on all nodes. So there is no need to localize them > from the node where job was submitted. I am wondering if we use the > SPARK_CLASSPATH approach, then it would not do localization. That would be > an added benefit. > Please clarify.
spark.executor.extraClassPath doesn't localize anything. It just prepends those classpath entries to the usual classpath used to launch the executor. There's no copying of files or anything, so they're expected to exist on the nodes. It's basically exactly the same as SPARK_CLASSPATH, but broken down to two options (one for the executors, and one for the driver). -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
