Hi,
When playing with spark, I encountered an ExceptionInInitializerError. I am
confused with the closure which spark uses.
For example, I have 2 scala files in my project:
*// Entry.scala
object Entry {
def main(args: Array[String]) = {
anonymous.run()
}
}
// anonymous.scala
object anonymous {
val rdd1 = sparkContext.parallelize(List(1, 2, 3, 4))
val a = 1
def run() {
val rdd2 = rdd1.map(_ + a)
println(rdd2.count)
}
}*
The code above will give an ExceptionInInitializerError which says:
*java.lang.ExceptionInInitializerError
at job.analysis.anonymous$.<init>(anonymous.scala:37)
at job.analysis.anonymous$.<clinit>(anonymous.scala)
...
java.lang.NoClassDefFoundError: Could not initialize class
job.analysis..anonymous$*
But if I move "*val a = 1*" into the *run() * function, it works. So I think
the problem is the object initialization.
The object "anonymous" here is only created in the main process, not on the
worker process, thus, when the workers want to access to "val a" which is
defined as an attribute of "anonymous". They just can not find it the
initialized object. Correct me if I am wrong.
I have also tested more about it:
*// snippet 1
def run() = {
class t extends Serializable {
val a = 2
}
val tt = new t()
val rdd2 = rdd1.map(_ + tt.a)
println(rdd2.count)
} // this one works
// snippet 2
def run() = {
object t extends Serializable {
val a = 2
}
val rdd2 = rdd1.map(_ + t.a)
println(rdd2.count)
} // I encountered the same ExceptionInInitializerError*
It seems strange to me, because, in snippet 1, the instance tt is created in
the main process the same as snippet 2, except,
in snippet 2, t is a standalone object.
Could someone explain why it occurs ? Maybe it's just a basic question, but
it really makes me confused a lot. Am I missing anything here ?
Hao
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/closure-and-ExceptionInInitializerError-tp77.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
