This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 8743571 [SPARK-33571][SQL][DOCS][3.0] Add a ref to INT96 config from
the doc for `spark.sql.legacy.parquet.datetimeRebaseModeInWrite/Read`
8743571 is described below
commit 874357174d8a2c9ef207a4dcd0ca819e658b2881
Author: Max Gekk <[email protected]>
AuthorDate: Fri Dec 4 15:10:51 2020 -0800
[SPARK-33571][SQL][DOCS][3.0] Add a ref to INT96 config from the doc for
`spark.sql.legacy.parquet.datetimeRebaseModeInWrite/Read`
### What changes were proposed in this pull request?
For the SQL configs `spark.sql.legacy.parquet.datetimeRebaseModeInWrite`
and `spark.sql.legacy.parquet.datetimeRebaseModeInRead`, improve their
descriptions by:
1. Explicitly document on which parquet types, those configs influence on
2. Refer to corresponding configs for `INT96`
### Why are the changes needed?
To avoid user confusions like reposted in SPARK-33571, and make the config
description more precise.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By running `./dev/scalastyle`.
Closes #30604 from MaxGekk/clarify-rebase-docs-3.0.
Authored-by: Max Gekk <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 23d1d70..0cf980c 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2537,7 +2537,9 @@ object SQLConf {
"to the legacy hybrid (Julian + Gregorian) calendar when writing
Parquet files. " +
"When CORRECTED, Spark will not do rebase and write the
dates/timestamps as it is. " +
"When EXCEPTION, which is the default, Spark will fail the writing if
it sees " +
- "ancient dates/timestamps that are ambiguous between the two
calendars.")
+ "ancient dates/timestamps that are ambiguous between the two
calendars. " +
+ "This config influences on writes of the following parquet logical
types: DATE, " +
+ "TIMESTAMP_MILLIS, TIMESTAMP_MICROS. Spark always performs rebasing
for the INT96 type.")
.version("3.0.0")
.stringConf
.transform(_.toUpperCase(Locale.ROOT))
@@ -2552,7 +2554,9 @@ object SQLConf {
"When CORRECTED, Spark will not do rebase and read the
dates/timestamps as it is. " +
"When EXCEPTION, which is the default, Spark will fail the reading if
it sees " +
"ancient dates/timestamps that are ambiguous between the two
calendars. This config is " +
- "only effective if the writer info (like Spark, Hive) of the Parquet
files is unknown.")
+ "only effective if the writer info (like Spark, Hive) of the Parquet
files is unknown. " +
+ "This config influences on reads of the following parquet logical
types: DATE, " +
+ "TIMESTAMP_MILLIS, TIMESTAMP_MICROS. Spark always performs rebasing
for the INT96 type.")
.version("3.0.0")
.stringConf
.transform(_.toUpperCase(Locale.ROOT))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
