You can copy the dependencies to all nodes in your cluster, and then
use "spark.{executor,driver}.extraClassPath" to add them to the
classpath of your executors / driver.On Mon, Apr 13, 2015 at 4:15 AM, Michael Weir <michael.weir....@gmail.com> wrote: > My app works fine with the single, "uber" jar file containing my app and all > its dependencies. However, it takes about 20 minutes to copy the 65MB jar > file up to the node on the cluster, so my "code, compile, test" cycle has > become a "core, compile, coooooopppppppyyyy, test" cycle. > > I'd like to have a single dependencies jar file on the node, and use a > separate small jar for my app (which takes around 10 seconds to copy to the > node). > > I've tried using "--jars deps.jar", but that copies the deps.jar to the > app-* folder but not the driver-* folder, so I get classNotFound errors on > the driver. Various other combinations of flags, etc. have produced a fair > bit of frustration but no progress. > > Any help with this would be greatly appreciated, as this problem is > significantly stretching the length of my work day! > > Thanks. -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
