Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22237#discussion_r212850156
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -469,4 +470,23 @@ class JsonFunctionsSuite extends QueryTest with
SharedSQLContext {
checkAnswer(sql("""select json[0] from jsonTable"""), Seq(Row(null)))
}
+
+ test("from_json invalid json - check modes") {
+ val df = Seq("""{"a" 1}""", """{"a": 2}""").toDS()
+ val schema = new StructType().add("a", IntegerType)
+
+ checkAnswer(
+ df.select(from_json($"value", schema, Map("mode" -> "PERMISSIVE"))),
+ Row(Row(null)) :: Row(Row(2)) :: Nil)
+
+ val exceptionOne = intercept[SparkException] {
+ df.select(from_json($"value", schema, Map("mode" ->
"FAILFAST"))).collect()
--- End diff --
`JsonToStructs` resembles PERMISSIVE mode (from the first place) although
their behaviours are slightly different. This is going to be different with
PERMISSIVE and also FAILFAST modes. They are actually behaviour changes if we
just use PERMISSIVE mode here by default (as @viirya pointed out).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
