At first glance that looks like an error with the class shipping in the
spark shell. (i.e. the line that you type into the spark shell are
compiled into classes and then shipped to the executors where they run).
Are you able to run other spark examples with closures in the same shell?
Michael
On Wed, Jul 9, 2014 at 4:28 AM, gil...@gmail.com <gil...@gmail.com> wrote:
> Hello, While trying to run this example below I am getting errors. I have
> build Spark using the followng command: $ SPARK_HADOOP_VERSION=2.4.0
> SPARK_YARN=true SPARK_HIVE=true sbt/sbt clean assembly
> --------------------------------------- --Running the example using
> Spark-shell --------------------------------------- $
> SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.1.0-SNAPSHOT-hadoop2.4.0.jar
> HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop MASTER=yarn-client
> ./bin/spark-shell scala>.... val sqlContext = new
> org.apache.spark.sql.SQLContext(sc) import sqlContext._ case class
> Person(name: String, age: Int) val people = sc.textFile("hdfs://
> myd-vm05698.hpswlabs.adapps.hp.com:9000/user/spark/examples/people.txt").map(_.split(",")).map(p
> => Person(p(0), p(1).trim.toInt)) people.registerAsTable("people") val
> teenagers = sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")
> teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
> ---------------------------------------------- error
> -----------------------------------------------
> java.lang.NoClassDefFoundError: Could not initialize class $line10.$read$
> at $line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at
> $line14.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(:19) at
> scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at
> scala.collection.Iterator$$anon$1.next(Iterator.scala:853) at
> scala.collection.Iterator$$anon$1.head(Iterator.scala:840) at
> org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:181)
> at
> org.apache.spark.sql.execution.ExistingRdd$$anonfun$productToRowRdd$1.apply(basicOperators.scala:176)
> at org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at
> org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:580) at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
> org.apache.spark.sql.SchemaRDD.compute(SchemaRDD.scala:112) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261) at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:228) at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:112) at
> org.apache.spark.scheduler.Task.run(Task.scala:51) at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> ------------------------------
> View this message in context: Spark SQL - java.lang.NoClassDefFoundError:
> Could not initialize class $line10.$read$
> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-java-lang-NoClassDefFoundError-Could-not-initialize-class-line10-read-tp9170.html>
> Sent from the Apache Spark User List mailing list archive
> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com.
>
