Re: Write to Cassandra table from pyspark fails with scala reflect error [RESOLVED]
There was some environment issue. I basically removed all environment variables
and with 2.11 and it worked.
Thanks for help.
From: Trivedi Amit <amit_...@yahoo.com.INVALID>
To: Russell Spitzer <russell.spit...@gmail.com>; "user@spark.apache.org"
<user@spark.apache.org>
Sent: Thursday, September 15, 2016 9:47 AM
Subject: Re: Write to Cassandra table from pyspark fails with scala reflect
error
Thanks Russell. I didn't build this myself. I tried with Scala 2.11
com.datastax.spark:spark-cassandra-connector_2.11:2.0.0-M(1-3) and I am getting
```
Exception in thread "main" java.lang.RuntimeException: [download failed:
org.scala-lang#scala-reflect;2.11.8!scala-reflect.jar]
at
org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1066)
at
org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:294)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:158)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
File "$SPARK_HOME/python/pyspark/shell.py", line 38, in
SparkContext._ensure_initialized()
File "$SPARK_HOME/python/pyspark/context.py", line 243, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway()
File "$SPARK_HOME/python/pyspark/java_gateway.py", line 94, in launch_gateway
raise Exception("Java gateway process exited before sending the driver its
port number")
Exception: Java gateway process exited before sending the driver its port
number```
I deleted my .m2 directory to avoid any conflicts with cached or older
versions. I only have SPARK_HOME environment variable set (env variables
related to Spark and Python).
From: Russell Spitzer <russell.spit...@gmail.com>
To: Trivedi Amit <amit_...@yahoo.com>; "user@spark.apache.org"
<user@spark.apache.org>
Sent: Wednesday, September 14, 2016 11:24 PM
Subject: Re: Write to Cassandra table from pyspark fails with scala reflect
error
Spark 2.0 defaults to Scala 2.11, so if you didn't build it yourself you need
the 2.11 artifact for the Spark Cassandra Connector.
On Wed, Sep 14, 2016 at 7:44 PM Trivedi Amit <amit_...@yahoo.com.invalid> wrote:
Hi,
I am testing a pyspark program that will read from a csv file and write data
into Cassandra table. I am using pyspark with spark-cassandra-connector
2.10:2.0.0-M3. I am using Spark v2.0.0.
While executing below command
```df.write.format("org.apache.spark.sql.cassandra").mode('append').options(
table="test_table", keyspace="test").save()```
I am getting
```
py4j.protocol.Py4JJavaError: An error occurred while calling o47.save.
: java.lang.NoSuchMethodError:
scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
at
com.datastax.spark.connector.types.TypeConverter$.(TypeConverter.scala:116)
at
com.datastax.spark.connector.types.TypeConverter$.(TypeConverter.scala)
at
com.datastax.spark.connector.types.BigIntType$.converterToCassandra(PrimitiveColumnType.scala:50)
at
com.datastax.spark.connector.types.BigIntType$.converterToCassandra(PrimitiveColumnType.scala:46)
at
com.datastax.spark.connector.writer.SqlRowWriter$$anonfun$3.apply(SqlRowWriter.scala:18)
at
com.datastax.spark.connector.writer.SqlRowWriter$$anonfun$3.apply(SqlRowWriter.scala:18)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
com.datastax.spark.connector.writer.SqlRowWriter.(SqlRowWriter.scala:18)
at
com.datastax.spark.connector.writer.SqlRowWriter$Factory$.rowWriter(SqlRowWriter.scala:36)
at
com.datastax.spark.connector.writer.SqlRowWriter$Factory$.rowWriter(SqlRowWriter.scala:34)
at
com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:271)
at
com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
at
org.apache.spark.sql.cassandra.CassandraSourceRelation.insert(CassandraSourceRelation.scala:66)
at
org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:85)
at
org.apache.spark.s
Re: Write to Cassandra table from pyspark fails with scala reflect error
Thanks Russell. I didn't build this myself. I tried with Scala 2.11
com.datastax.spark:spark-cassandra-connector_2.11:2.0.0-M(1-3) and I am getting
```
Exception in thread "main" java.lang.RuntimeException: [download failed:
org.scala-lang#scala-reflect;2.11.8!scala-reflect.jar]
at
org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1066)
at
org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:294)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:158)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
File "$SPARK_HOME/python/pyspark/shell.py", line 38, in
SparkContext._ensure_initialized()
File "$SPARK_HOME/python/pyspark/context.py", line 243, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway()
File "$SPARK_HOME/python/pyspark/java_gateway.py", line 94, in launch_gateway
raise Exception("Java gateway process exited before sending the driver its
port number")
Exception: Java gateway process exited before sending the driver its port
number```
I deleted my .m2 directory to avoid any conflicts with cached or older
versions. I only have SPARK_HOME environment variable set (env variables
related to Spark and Python).
From: Russell Spitzer <russell.spit...@gmail.com>
To: Trivedi Amit <amit_...@yahoo.com>; "user@spark.apache.org"
<user@spark.apache.org>
Sent: Wednesday, September 14, 2016 11:24 PM
Subject: Re: Write to Cassandra table from pyspark fails with scala reflect
error
Spark 2.0 defaults to Scala 2.11, so if you didn't build it yourself you need
the 2.11 artifact for the Spark Cassandra Connector.
On Wed, Sep 14, 2016 at 7:44 PM Trivedi Amit <amit_...@yahoo.com.invalid> wrote:
Hi,
I am testing a pyspark program that will read from a csv file and write data
into Cassandra table. I am using pyspark with spark-cassandra-connector
2.10:2.0.0-M3. I am using Spark v2.0.0.
While executing below command
```df.write.format("org.apache.spark.sql.cassandra").mode('append').options(
table="test_table", keyspace="test").save()```
I am getting
```
py4j.protocol.Py4JJavaError: An error occurred while calling o47.save.
: java.lang.NoSuchMethodError:
scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
at
com.datastax.spark.connector.types.TypeConverter$.(TypeConverter.scala:116)
at
com.datastax.spark.connector.types.TypeConverter$.(TypeConverter.scala)
at
com.datastax.spark.connector.types.BigIntType$.converterToCassandra(PrimitiveColumnType.scala:50)
at
com.datastax.spark.connector.types.BigIntType$.converterToCassandra(PrimitiveColumnType.scala:46)
at
com.datastax.spark.connector.writer.SqlRowWriter$$anonfun$3.apply(SqlRowWriter.scala:18)
at
com.datastax.spark.connector.writer.SqlRowWriter$$anonfun$3.apply(SqlRowWriter.scala:18)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
com.datastax.spark.connector.writer.SqlRowWriter.(SqlRowWriter.scala:18)
at
com.datastax.spark.connector.writer.SqlRowWriter$Factory$.rowWriter(SqlRowWriter.scala:36)
at
com.datastax.spark.connector.writer.SqlRowWriter$Factory$.rowWriter(SqlRowWriter.scala:34)
at
com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:271)
at
com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
at
org.apache.spark.sql.cassandra.CassandraSourceRelation.insert(CassandraSourceRelation.scala:66)
at
org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:85)
at
org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.
Re: Write to Cassandra table from pyspark fails with scala reflect error
Hi,
I am testing a pyspark program that will read from a csv file and write data
into Cassandra table. I am using pyspark with spark-cassandra-connector
2.10:2.0.0-M3. I am using Spark v2.0.0.
While executing below command
```df.write.format("org.apache.spark.sql.cassandra").mode('append').options(
table="test_table", keyspace="test").save()```
I am getting
```
py4j.protocol.Py4JJavaError: An error occurred while calling o47.save.
: java.lang.NoSuchMethodError:
scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
at
com.datastax.spark.connector.types.TypeConverter$.(TypeConverter.scala:116)
at
com.datastax.spark.connector.types.TypeConverter$.(TypeConverter.scala)
at
com.datastax.spark.connector.types.BigIntType$.converterToCassandra(PrimitiveColumnType.scala:50)
at
com.datastax.spark.connector.types.BigIntType$.converterToCassandra(PrimitiveColumnType.scala:46)
at
com.datastax.spark.connector.writer.SqlRowWriter$$anonfun$3.apply(SqlRowWriter.scala:18)
at
com.datastax.spark.connector.writer.SqlRowWriter$$anonfun$3.apply(SqlRowWriter.scala:18)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
com.datastax.spark.connector.writer.SqlRowWriter.(SqlRowWriter.scala:18)
at
com.datastax.spark.connector.writer.SqlRowWriter$Factory$.rowWriter(SqlRowWriter.scala:36)
at
com.datastax.spark.connector.writer.SqlRowWriter$Factory$.rowWriter(SqlRowWriter.scala:34)
at
com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:271)
at
com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:36)
at
org.apache.spark.sql.cassandra.CassandraSourceRelation.insert(CassandraSourceRelation.scala:66)
at
org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:85)
at
org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:211)
at java.lang.Thread.run(Thread.java:745)
```
Google search for this error lead to threads where folks were talking about
using scala version 2.10 instead of 2.11 for this issue. However, I am not
using Scala and I am assuming that 2.10 in spark-cassandra-connector is Scala
version.
Don't know how to fix or get around this issue. Appreciate any help.
ThanksAT
