Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21741#discussion_r202451374
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -378,6 +378,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_FILTER_PUSHDOWN_TIMESTAMP_ENABLED =
+ buildConf("spark.sql.parquet.filterPushdown.timestamp")
+ .doc("If true, enables Parquet filter push-down optimization for
Timestamp. " +
+ "This configuration only has an effect when
'spark.sql.parquet.filterPushdown' is " +
+ "enabled and Timestamp stored as TIMESTAMP_MICROS or
TIMESTAMP_MILLIS type.")
+ .internal()
+ .booleanConf
+ .createWithDefault(true)
--- End diff --
Because we're using the file schema, it doesn't mater what the write
configuration is. It only matters what it was when the file was written. If the
file has an INT96 timestamp, this should just not push anything down.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
