Cheng Lian created SPARK-10588:
----------------------------------
Summary: Saving a DataFrame containing only nulls to JSON doesn't
work
Key: SPARK-10588
URL: https://issues.apache.org/jira/browse/SPARK-10588
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0
Reporter: Cheng Lian
Snippets to reproduce this issue:
{noformat}
val path = "file:///tmp/spark/null"
// A single row containing a single null double, saving to JSON, wrong
sqlContext.
range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0").
write.mode("overwrite").json(path)
sqlContext.read.json(path).show()
++
||
++
||
++
// Two rows each containing a single null double, saving to JSON, wrong
sqlContext.
range(2).selectExpr("CAST(NULL AS DOUBLE) AS c0").
write.mode("overwrite").json(path)
sqlContext.read.json(path).show()
++
||
++
||
||
++
// A single row containing two null doubles, saving to JSON, wrong
sqlContext.
range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0", "CAST(NULL AS DOUBLE) AS
c1").
write.mode("overwrite").json(path)
sqlContext.read.json(path).show()
++
||
++
||
++
// A single row containing a single null double, saving to Parquet, OK
sqlContext.
range(1).selectExpr("CAST(NULL AS DOUBLE) AS c0").
write.mode("overwrite").parquet(path)
sqlContext.read.parquet(path).show()
+----+
| d|
+----+
|null|
+----+
// Two rows, one containing a single null double, one containing non-null
double, saving to JSON, OK
sqlContext.
range(2).selectExpr("IF(id % 2 = 0, CAST(NULL AS DOUBLE), id) AS c0").
write.mode("overwrite").json(path)
sqlContext.read.json(path).show()
+----+
| d|
+----+
|null|
| 1.0|
+----+
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
