Peter Mead created SPARK-20252:
----------------------------------
Summary: java.lang.ClassNotFoundException:
$line22.$read$$iwC$$iwC$movie_row
Key: SPARK-20252
URL: https://issues.apache.org/jira/browse/SPARK-20252
Project: Spark
Issue Type: Bug
Components: Spark Shell
Affects Versions: 1.6.3
Environment: Datastax DSE dual node SPARK cluster
[cqlsh 5.0.1 | Cassandra 3.0.12.1586 | DSE 5.0.7 | CQL spec 3.4.0 | Native
protocol v4]
Reporter: Peter Mead
After starting a spark shell using DSE -u xxxxxxxx -p xxxxxxxxx spark
scala> case class movie_row (actor: String, character_name: String, video_id:
java.util.UUID, video_year: Int, title: String)
defined class movie_row
scala> val vids=sc.cassandraTable("hcl","videos_by_actor")
vids:
com.datastax.spark.connector.rdd.CassandraTableScanRDD[com.datastax.spark.connector.CassandraRow]
= CassandraTableScanRDD[0] at RDD at CassandraRDD.scala:15
scala> val vids=sc.cassandraTable[movie_row]("hcl","videos_by_actor")
vids: com.datastax.spark.connector.rdd.CassandraTableScanRDD[movie_row] =
CassandraTableScanRDD[1] at RDD at CassandraRDD.scala:15
scala> vids.count
res0: Long = 114961
Works OK!!
BUT if the spark context is stopped and recreated THEN:
scala> sc.stop()
scala> import org.apache.spark.SparkContext, org.apache.spark.SparkContext._,
org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
scala> :paste
// Entering paste mode (ctrl-D to finish)
val conf = new SparkConf(true)
.set("spark.cassandra.connection.host", "redacted")
.set("spark.cassandra.auth.username", "redacted")
.set("spark.cassandra.auth.password", "redacted")
// Exiting paste mode, now interpreting.
conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@e207342
scala> val sc = new SparkContext("spark://192.168.1.84:7077", "pjm", conf)
sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@12b8b7c8
scala> case class movie_row (actor: String, character_name: String, video_id:
java.util.UUID, video_year: Int, title: String)
defined class movie_row
scala> val vids=sc.cassandraTable[movie_row]("hcl","videos_by_actor")
vids: com.datastax.spark.connector.rdd.CassandraTableScanRDD[movie_row] =
CassandraTableScanRDD[0] at RDD at CassandraRDD.scala:15
scala> vids.count
[Stage 0:> (0 + 2) /
2]WARN 2017-04-07 12:52:03,277 org.apache.spark.scheduler.TaskSetManager:
Lost task 0.0 in stage 0.0 (TID 0, cassandra183):
java.lang.ClassNotFoundException: $line22.$read$$iwC$$iwC$movie_row
FAILS!!
I have been unable to get this to work from a remote SPARK shell!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
