This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a67acbaa29d [SPARK-37575][SQL][FOLLOWUP] Update the migration guide
for added legacy flag for the breaking change of write null value in csv to
unquoted empty string
a67acbaa29d is described below
commit a67acbaa29d1ab9071910cac09323c2544d65303
Author: Xinyi Yu <[email protected]>
AuthorDate: Wed Apr 20 10:48:00 2022 +0800
[SPARK-37575][SQL][FOLLOWUP] Update the migration guide for added legacy
flag for the breaking change of write null value in csv to unquoted empty string
### What changes were proposed in this pull request?
This is a follow-up of updating the migration guide for
https://github.com/apache/spark/pull/36110 which adds a legacy flag to restore
the pre-change behavior.
It also fixes a typo in the previous flag description.
### Why are the changes needed?
The flag needs to be documented.
### Does this PR introduce _any_ user-facing change?
It changes the migration doc for users.
### How was this patch tested?
No tests
Closes #36268 from anchovYu/flags-null-to-csv-migration-guide.
Authored-by: Xinyi Yu <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
docs/sql-migration-guide.md | 2 +-
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index c4f1bd188bf..32b90da1917 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -54,7 +54,7 @@ license: |
- Since Spark 3.3, the `strfmt` in `format_string(strfmt, obj, ...)` and
`printf(strfmt, obj, ...)` will no longer support to use "0$" to specify the
first argument, the first argument should always reference by "1$" when use
argument index to indicating the position of the argument in the argument list.
- - Since Spark 3.3, nulls are written as empty strings in CSV data source by
default. In Spark 3.2 or earlier, nulls were written as empty strings as quoted
empty strings, `""`. To restore the previous behavior, set `nullValue` to `""`.
+ - Since Spark 3.3, nulls are written as empty strings in CSV data source by
default. In Spark 3.2 or earlier, nulls were written as empty strings as quoted
empty strings, `""`. To restore the previous behavior, set `nullValue` to `""`,
or set the configuration
`spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv` to `true`.
- Since Spark 3.3, DESCRIBE FUNCTION fails if the function does not exist.
In Spark 3.2 or earlier, DESCRIBE FUNCTION can still run and print "Function:
func_name not found".
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 36b666fd59c..301d792bb3e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -3758,7 +3758,7 @@ object SQLConf {
buildConf("spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv")
.internal()
.doc("When set to false, nulls are written as unquoted empty strings in
CSV data source. " +
- "If set to false, it restores the legacy behavior that nulls were
written as quoted " +
+ "If set to true, it restores the legacy behavior that nulls were
written as quoted " +
"empty strings, `\"\"`.")
.version("3.3.0")
.booleanConf
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
