Github user fuqiliang commented on the issue:
https://github.com/apache/spark/pull/20666
for specify, the json file (Sanity4.json) is
`{"a":"a1","int":1,"other":4.4}
{"a":"a2","int":"","other":""}`
code ï¼
> val config = new SparkConf().setMaster("local[5]").setAppName("test")
> val sc = SparkContext.getOrCreate(config)
> val sql = new SQLContext(sc)
>
> val file_path =
this.getClass.getClassLoader.getResource("Sanity4.json").getFile
> val df = sql.read.schema(null).json(file_path)
> df.show(30)
then in spark 1.6, result is
+---+----+-----+
| a| int|other|
+---+----+-----+
| a1| 1| 4.4|
| a2|null| null|
+---+----+-----+
root
|-- a: string (nullable = true)
|-- int: long (nullable = true)
|-- other: double (nullable = true)
but in spark 2.2, result is
+----+----+-----+
| a| int|other|
+----+----+-----+
| a1| 1| 4.4|
|null|null| null|
+----+----+-----+
root
|-- a: string (nullable = true)
|-- int: long (nullable = true)
|-- other: double (nullable = true)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
