Thanks for your response and suggestion, Sean. Setting
"spark.files.userClassPathFirst" didn't fix the problem for me. I am not
very familiar with the Spark and Scala environment, so please correct any
incorrect assumptions or statements I make.
However, I don't believe this to be a classpath visibility issue. I wrote a
small helper method to print out the classpath from both the driver and
worker, and the output is identical. (I'm printing out
System.getProperty("java.class.path") -- is there a better way to do this
or check the class path?)
Furthermore, userClassPathFirst seems to have a detrimental effect on
otherwise working code, which I cannot explain or do not understand.
For example, I created a trivial RDD as such:
val l = List(1, 2, 3)
sc.makeRDD(l).foreach((x: Int) => {
println(x.toString)
})
With userClassPathFirst set, I encounter a java.lang.ClassCastException
trying to execute that code. Is that to be expected?
I created a standalone repository with a very simple Spark job that
exhibits the above issues. I'd appreciate if you could take a look.
https://github.com/vibhav/spark-protobuf
On Tue, Feb 24, 2015 at 2:06 AM Sean Owen <so...@cloudera.com> wrote:
> I assume this is a difference between your local driver classpath and
> remote worker classpath. It may not be a question of whether the class
> is there, but classpath visibility issues. Have you looked into
> settings like spark.files.userClassPathFirst?
>
> On Tue, Feb 24, 2015 at 4:43 AM, necro351 . <necro...@gmail.com> wrote:
> > Hello,
> >
> > I am trying to deserialize some data encoded using proto buff from within
> > Spark and am getting class-not-found exceptions. I have narrowed the
> program
> > down to something very simple that shows the problem exactly (see 'The
> > Program' below) and hopefully someone can tell me the easy fix :)
> >
>
