Hi Geoff -

Appreciate the help here - I do understand what you’re saying below.  And I am 
able to get this working when I submit a job to a local cluster.

I think part of the issue here is that there’s ambiguity in the terminology.  
When I say “LOCAL” spark, I mean an instance of spark that is created by my 
driver program, and is not a cluster itself.  It means that my master node is 
“local”, and this mode is primarily used for testing.


While I am able to get alluxio working with spark-submit, I am unable to get it 
working when using local mode.  The mechanisms for setting class paths during 
spark-submit are not available in local mode.  My understanding is that all one 
is able to use is:


To set any runtime properties of the local instance.  Note that it is possible 
(and I am more convinced of this as time goes on) that alluxio simply does not 
work in spark local mode as described above.

> On Apr 13, 2018, at 11:09 AM, Geoff Von Allmen <ge...@ibleducation.com> wrote:
> I fought with a ClassNotFoundException for quite some time, but it was for 
> kafka.
> The final configuration that got everything working was running spark-submit 
> with the following options:
> --jars "/path/to/.ivy2/jars/package.jar" \
> --driver-class-path "/path/to/.ivy2/jars/package.jar" \
> --conf "spark.executor.extraClassPath=/path/to/.ivy2/package.jar" \
> --packages org.some.package:package_name:version
> While this was needed for me to run in cluster mode, it works equally well 
> for client mode as well.
> One other note when needing to supplied multiple items to these args - --jars 
> and --packages should be comma separated, --driver-class-path and 
> extraClassPath should be : separated
> On Fri, Apr 13, 2018 at 4:28 AM, jb44 <jbo...@gmail.com 
> <mailto:jbo...@gmail.com>> wrote:
> Haoyuan -
> As I mentioned below, I've been through the documentation already.  It has
> not helped me to resolve the issue.
> Here is what I have tried so far:
> - setting extraClassPath as explained below
> - adding fs.alluxio.impl through sparkconf
> - adding spark.sql.hive.metastore.sharedPrefixes (though I don't believe
> this matters in my case)
> - compiling the client from source 
> Do you have any other suggestions on how to get this working?  
> Thanks
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ 
> <http://apache-spark-user-list.1001560.n3.nabble.com/>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org 
> <mailto:user-unsubscr...@spark.apache.org>

Reply via email to