Ok - `LOCAL` makes sense now. Do you have the option to still use `spark-submit` in this scenario, but using the following options:
```bash --master "local[*]" \ --deploy-mode "client" \ ... ``` I know in the past, I have setup some options using `.config("Option", "value")` when creating the spark session, and then other runtime options as you describe above with `spark.conf.set`. At this point though I've just moved everything out into a `spark-submit` script. On Fri, Apr 13, 2018 at 8:18 AM, Jason Boorn <jbo...@gmail.com> wrote: > Hi Geoff - > > Appreciate the help here - I do understand what you’re saying below. And > I am able to get this working when I submit a job to a local cluster. > > I think part of the issue here is that there’s ambiguity in the > terminology. When I say “LOCAL” spark, I mean an instance of spark that is > created by my driver program, and is not a cluster itself. It means that > my master node is “local”, and this mode is primarily used for testing. > > https://jaceklaskowski.gitbooks.io/mastering-apache- > spark/content/spark-local.html > > While I am able to get alluxio working with spark-submit, I am unable to > get it working when using local mode. The mechanisms for setting class > paths during spark-submit are not available in local mode. My > understanding is that all one is able to use is: > > spark.conf.set(“”) > > To set any runtime properties of the local instance. Note that it is > possible (and I am more convinced of this as time goes on) that alluxio > simply does not work in spark local mode as described above. > > > On Apr 13, 2018, at 11:09 AM, Geoff Von Allmen <ge...@ibleducation.com> > wrote: > > I fought with a ClassNotFoundException for quite some time, but it was > for kafka. > > The final configuration that got everything working was running > spark-submit with the following options: > > --jars "/path/to/.ivy2/jars/package.jar" \ > --driver-class-path "/path/to/.ivy2/jars/package.jar" \ > --conf "spark.executor.extraClassPath=/path/to/.ivy2/package.jar" \ > --packages org.some.package:package_name:version > > While this was needed for me to run in cluster mode, it works equally > well for client mode as well. > > One other note when needing to supplied multiple items to these args - > --jars and --packages should be comma separated, --driver-class-path and > extraClassPath should be : separated > > HTH > > > On Fri, Apr 13, 2018 at 4:28 AM, jb44 <jbo...@gmail.com> wrote: > >> Haoyuan - >> >> As I mentioned below, I've been through the documentation already. It has >> not helped me to resolve the issue. >> >> Here is what I have tried so far: >> >> - setting extraClassPath as explained below >> - adding fs.alluxio.impl through sparkconf >> - adding spark.sql.hive.metastore.sharedPrefixes (though I don't believe >> this matters in my case) >> - compiling the client from source >> >> Do you have any other suggestions on how to get this working? >> >> Thanks >> >> >> >> -- >> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> > >