Wenchen Fan created SPARK-17725:
-----------------------------------
Summary: Spark should not write out parquet files with schema
containing non-nullable fields
Key: SPARK-17725
URL: https://issues.apache.org/jira/browse/SPARK-17725
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.0
Reporter: Wenchen Fan
Since Spark 1.3, after PR https://github.com/apache/spark/pull/4826 , Spark SQL
will always set all schema fields to nullable before writing out parquet files,
to make the data pipeline more robust.
However, this behaviour has been changed in 2.0 accidently by PR
https://github.com/apache/spark/pull/11509
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
