Repository: spark Updated Branches: refs/heads/branch-1.6 c42433d02 -> 523db0df5
[SPARK-11360][DOC] Loss of nullability when writing parquet files This fix is to add one line to explain the current behavior of Spark SQL when writing Parquet files. All columns are forced to be nullable for compatibility reasons. Author: gatorsmile <[email protected]> Closes #9314 from gatorsmile/lossNull. (cherry picked from commit 2f38378856fb56bdd9be7ccedf56427e81701f4e) Signed-off-by: Michael Armbrust <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/523db0df Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/523db0df Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/523db0df Branch: refs/heads/branch-1.6 Commit: 523db0df52f7c3a9c90192e1ac33e7efd0cfa701 Parents: c42433d Author: gatorsmile <[email protected]> Authored: Mon Nov 9 16:06:48 2015 -0800 Committer: Michael Armbrust <[email protected]> Committed: Mon Nov 9 16:07:21 2015 -0800 ---------------------------------------------------------------------- docs/sql-programming-guide.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/523db0df/docs/sql-programming-guide.md ---------------------------------------------------------------------- diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index ccd2690..6e02d65 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -982,7 +982,8 @@ when a table is dropped. [Parquet](http://parquet.io) is a columnar format that is supported by many other data processing systems. Spark SQL provides support for both reading and writing Parquet files that automatically preserves the schema -of the original data. +of the original data. When writing Parquet files, all columns are automatically converted to be nullable for +compatibility reasons. ### Loading Data Programmatically --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
