Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19702#discussion_r150131141
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1143,6 +1159,18 @@ class SQLConf extends Serializable with Logging {
def isParquetINT64AsTimestampMillis: Boolean =
getConf(PARQUET_INT64_AS_TIMESTAMP_MILLIS)
+ def parquetOutputTimestampType: ParquetOutputTimestampType.Value = {
+ val isOutputTimestampTypeSet =
settings.containsKey(PARQUET_OUTPUT_TIMESTAMP_TYPE.key)
+ if (!isOutputTimestampTypeSet && isParquetINT64AsTimestampMillis) {
+ // If PARQUET_OUTPUT_TIMESTAMP_TYPE is not set and
PARQUET_INT64_AS_TIMESTAMP_MILLIS is set,
+ // respect PARQUET_INT64_AS_TIMESTAMP_MILLIS and use
TIMESTAMP_MILLIS. Otherwise,
+ // PARQUET_OUTPUT_TIMESTAMP_TYPE has higher priority.
--- End diff --
BTW, do we have a simple test for this priority? seems
`isParquetINT64AsTimestampMillis` defaults to `false`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
