HyukjinKwon commented on a change in pull request #28137: [SPARK-31361][SQL]
Rebase datetime in parquet/avro according to file written Spark version
URL: https://github.com/apache/spark/pull/28137#discussion_r404673165
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2522,19 +2522,6 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
- val LEGACY_PARQUET_REBASE_DATETIME_IN_WRITE =
- buildConf("spark.sql.legacy.parquet.rebaseDateTimeInWrite.enabled")
Review comment:
One thing I am concerned, though, I suspect it won't be compatible with
other external projects if we don't rebase? We have
`spark.sql.parquet.writeLegacyFormat` in particular for the compatibility with
Apache Hive and Apache Impala for example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
