Hi, We have a Scala application and we want it to programmatically submit Spark jobs to a Spark-YARN cluster in yarn-client mode.
We're running into a lot of classpath issues, e.g. once submitted it looks for jars in our parent Scala application's local directory, jars that it shouldn't need. Our setJars in the SparkContext only mentions our fat jar, which should be all it needs. We are not sure why the other jars are being included once we submit and we don't see a mechanism to control what it wants. Here's a sample error: Diagnostics: java.io.FileNotFoundException: File file:/Users/github/spark/kindling-container/lib/spark-assembly-1.2.1-hadoop2.4.0.jar does not exist Failing this attempt. Failing the application. I read through the user list and there was discussion around possibly using Client.scala? Are there any code examples out there that we could use as reference? thanks, Michal
