Hi all,
TLDR: running spark locally through IntelliJ IDEA Scala Console results
in java.lang.ClassNotFoundException
Long version:
I'm an algorithms developer in SupersonicAds - an ad network. We are
building a major new big data project and we are now in the process of
selecting our tech stack & tools.
I'm new to Spark, but I'm very excited about it. It is my opinion that
Spark can be a great tool for us, and that we might be able to build most
of our toolchain on top of it.
We currently develop in Scala and we are using IntelliJ IDEA as our IDE (we
love it). One of the features I love about IDEA is the Scala Console which
lets me work interactively with all of my project's code available and all
of the IDE's features & convenience. That is as opposed to the Scala Shell
& Spark Shell which I dislike because it is based on JLine and doesn't
behave like a good shell would (I cant even Ctrl-c to abort a line without
crashing the whole thing). Of course, as an algo guy, having a good REPL is
crucial to me.
To get started, I added the following line to build.sbt:
> "org.apache.spark" %% "spark-core" % "0.9.1"
Then, added the following main class:
import org.apache.spark.SparkContext
> import org.apache.spark.SparkContext._
>
object Main extends App {
> val sc = new SparkContext("local", "myApp")
> val r = sc.parallelize(1 to 1000)
> println("r.filter(_ % 2 == 1).first() = " + r.filter(_ % 2 == 1).first())
> println("r.filter(_ % 2 == 1).count() = " + r.filter(_ % 2 == 1).count())
> }
Make, Run, Works perfectly.
Next, I try running the same in the scala console.
Bad news - the last line throws an exception:
> ERROR executor.Executor: Exception in task ID 0
> java.lang.ClassNotFoundException: $line5.$read$$iw$$iw$$iw$$iw$$anonfun$2
It is my guess that for some reason Spark is not able to find the anonymous
function (_ % 2 == 1). Please note I'm running locally so I did not provide
any jars. For some reason when using first() instead of count() it works.
Needless to say it also works in Spark Shell but as I stated, working with
it is not an option.
This issue brings much sadness to my heart, and I could not find a solution
on the mailing list archives or elsewhere. I am hoping someone here might
offer some help.
Thanks,
Jon
