Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/6299#discussion_r30766132
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
---
@@ -1074,4 +1073,16 @@ class JsonSuite extends QueryTest {
assert(StructType(Seq()) === emptySchema)
}
+ test("SPARK-7565 MapType in JsonRDD") {
+ val schemaWithSimpleMap = StructType(
+ StructField("map", MapType(StringType, IntegerType, true), false) ::
Nil)
+ val df = read.schema(schemaWithSimpleMap).json(mapType1)
+ val temp = Utils.createTempDir()
+ df.write.mode("overwrite").parquet(temp.getPath)
+
+ setConf("spark.sql.json.useJacksonStreamingAPI", "false")
--- End diff --
Since we are using a singleton `TestSQLContext`, this setting will also
affect other tests. How about we get the old value at the beginning of the test
and set the old value back at the end of the test? Also, let's also use
`checkAnswer` to make sure the data we read form the parquet data is correct.
btw, it will be good to test corrupt record column as well. We can have a
bad JSON record and then ask Spark SQL to infer the schema. Then, in the
schema, there will be a column called `_corrupt_record`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
