Matthew Farrellee created SPARK-6245:
----------------------------------------
Summary: jsonRDD() of empty RDD results in exception
Key: SPARK-6245
URL: https://issues.apache.org/jira/browse/SPARK-6245
Project: Spark
Issue Type: Bug
Components: PySpark, SQL, Streaming
Affects Versions: 1.2.1
Reporter: Matthew Farrellee
converting an empty RDD to a JSON RDD results in an exception. this case is
common when using spark streaming.
{code}
from pyspark import SparkContext
from pyspark.sql import SQLContext
sc = SparkContext()
qsc = SQLContext(sc)
qsc.jsonRDD(sc.parallelize([]))
{code}
exception:
{noformat}
Traceback (most recent call last):
File "/tmp/bug.py", line 5, in <module>
qsc.jsonRDD(sc.parallelize([]))
File "/usr/share/spark/python/pyspark/sql.py", line 1605, in jsonRDD
srdd = self._ssql_ctx.jsonRDD(jrdd.rdd(), samplingRatio)
File "/usr/share/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
line 538, in __call__
File "/usr/share/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py",
line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o27.jsonRDD.
: java.lang.UnsupportedOperationException: empty collection
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:886)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:886)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:886)
at org.apache.spark.sql.json.JsonRDD$.inferSchema(JsonRDD.scala:57)
at org.apache.spark.sql.SQLContext.jsonRDD(SQLContext.scala:232)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:207)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
