Re: Which directory contains third party libraries for Spark

Hey Stephen, In case these libraries exist on the client as a form of maven library, you can use --packages to ship the library and all it's dependencies, without building an uber jar. Best, Burak On Tue, Jul 28, 2015 at 10:23 AM, Marcelo Vanzin wrote: > Hi Stephen, > > There is no such direct

Re: Which directory contains third party libraries for Spark

Hi Stephen, There is no such directory currently. If you want to add an existing jar to every app's classpath, you need to modify two config values: spark.driver.extraClassPath and spark.executor.extraClassPath. On Mon, Jul 27, 2015 at 10:22 PM, Stephen Boesch wrote: > when using spark-submit:

Re: Which directory contains third party libraries for Spark

Can you show us the snippet of the exception stack ? Thanks > On Jul 27, 2015, at 10:22 PM, Stephen Boesch wrote: > > when using spark-submit: which directory contains third party libraries that > will be loaded on each of the slaves? I would like to scp one or more > libraries to each of t

Which directory contains third party libraries for Spark

when using spark-submit: which directory contains third party libraries that will be loaded on each of the slaves? I would like to scp one or more libraries to each of the slaves instead of shipping the contents in the application uber-jar. Note: I did try adding to $SPARK_HOME/lib_managed/jars.