Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/20637
The failure of
`org.apache.spark.sql.catalyst.expressions.JsonExpressionsSuite.from_json
missing fields` is due to passing `null` while the schema has `nullable=false`.
This inconsistency is agreed in the discussion at
[SPARK-23173](https://issues.apache.org/jira/browse/SPARK-23173).
`Assume that each field in schema passed to from_json is nullable, and
ignore the nullability information set in the passed schema.`
When `spark.sql.fromJsonForceNullableSchema=false`, I think that a test is
invalid to pass `nullable=false` in the corresponding schema to the missing
field.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
