Ok thanks - I was basing my design on this:
https://databricks.com/blog/2016/08/15/how-to-use-sparksession-in-apache-spark-2-0.html
<https://databricks.com/blog/2016/08/15/how-to-use-sparksession-in-apache-spark-2-0.html>
Wherein it says:
Once the SparkSession is instantiated, you can configure Spark’s runtime config
properties.
Apparently the suite of runtime configs you can change does not include
classpath.
So the answer to my original question is basically this:
When using local (pseudo-cluster) mode, there is no way to add external jars to
the spark instance. This means that Alluxio will not work with Spark when
Spark is run in master=local mode.
Thanks again - often getting a definitive “no” is almost as good as a yes.
Almost ;)
> On Apr 13, 2018, at 1:21 PM, Marcelo Vanzin <van...@cloudera.com> wrote:
>
> There are two things you're doing wrong here:
>
> On Thu, Apr 12, 2018 at 6:32 PM, jb44 <jbo...@gmail.com> wrote:
>> Then I can add the alluxio client library like so:
>> sparkSession.conf.set("spark.driver.extraClassPath", ALLUXIO_SPARK_CLIENT)
>
> First one, you can't modify JVM configuration after it has already
> started. So this line does nothing since it can't re-launch your
> application with a new JVM.
>
>> sparkSession.conf.set("spark.executor.extraClassPath", ALLUXIO_SPARK_CLIENT)
>
> There is a lot of configuration that you cannot set after the
> application has already started. For example, after the session is
> created, most probably this option will be ignored, since executors
> will already have started.
>
> I'm not so sure about what happens when you use dynamic allocation,
> but these post-hoc config changes in general are not expected to take
> effect.
>
> The documentation could be clearer about this (especially stuff that
> only applies to spark-submit), but that's the gist of it.
>
>
> --
> Marcelo
