MaxGekk opened a new pull request #31576:
URL: https://github.com/apache/spark/pull/31576
### What changes were proposed in this pull request?
Move the datetime rebase SQL configs from the `legacy` namespace by:
1. Renaming of the existing rebase configs like
`spark.sql.legacy.parquet.datetimeRebaseModeInRead` ->
`spark.sql.parquet.datetimeRebaseModeInRead`.
2. Add the legacy configs as alternatives
3. Deprecate the legacy rebase configs.
### Why are the changes needed?
The rebasing SQL configs like
`spark.sql.legacy.parquet.datetimeRebaseModeInRead` can be used not only for
migration from previous Spark versions but also to read/write datatime columns
saved by other systems/frameworks/libs. So, the configs shouldn't be considered
as legacy configs.
### Does this PR introduce _any_ user-facing change?
Should not. Users will see a warning if they still use one of the legacy
configs.
### How was this patch tested?
1. Manually checking new configs:
```scala
scala> spark.conf.get("spark.sql.parquet.datetimeRebaseModeInRead")
res0: String = EXCEPTION
scala> spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInRead",
"LEGACY")
21/02/17 14:57:10 WARN SQLConf: The SQL config
'spark.sql.legacy.parquet.datetimeRebaseModeInRead' has been deprecated in
Spark v3.2 and may be removed in the future. Use
'spark.sql.parquet.datetimeRebaseModeInRead' instead.
scala> spark.conf.get("spark.sql.parquet.datetimeRebaseModeInRead")
res2: String = LEGACY
```
2. By running a datetime rebasing test suite:
```
$ build/sbt "test:testOnly *ParquetRebaseDatetimeV1Suite"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
