alamb opened a new issue, #25484: URL: https://github.com/apache/datafusion/issues/25484
### Is your feature request related to a problem or challenge? The Parquet specification historically left the sort order of `INT96` undefined, so DataFusion doesn't trust `min`/`max` statistics of `INT96` columns (TODO code ref) and therefore cannot prune row groups, pages, or files using predicates on those columns. However, Parquet files written by Spark, Hive, Impala, and Databricks Photon commonly store timestamps as the (deprecated) `INT96` physical type and show no signs of changing this behavior. :thumbsdown: This means any query with a timestamp filter over a Spark-written dataset scans every row group, for example, using this file; TODO FILE ```sql ```` Note there is no pruning I made the file like this <details> ```shell ... ``` </details> The Parquet specification recently added a new `ColumnOrder` variant, `INT96_TIMESTAMP_ORDER`, in [apache/parquet-format#584]: https://github.com/apache/parquet-format/blob/2076361bb64e2de9ca6a8d06eda025a6fa4e9df6/src/main/thrift/parquet.thrift#L1230-L1233 to explicitly define the order arrow-rs / parquet `60.0.0` implemented this in - https://github.com/apache/arrow-rs/pull/10106 However, to avoid introducing any new correctness bugs, when upgrading to arrow/parquet `60.0.0` in https://github.com/apache/datafusion/pull/25335, I changed `has_untrusted_min_max_order` to keep treating `INT96` columns as untrusted (see the review comment at https://github.com/apache/datafusion/pull/25335/changes#r4018522067 and the code) However, this means that DataFusion will *still* not prune files with `INT96` , even those that could - File-level statistics: https://github.com/apache/datafusion/blob/5a502dffce2f1881479e8c1a5b68a4a5108e7934/datafusion/datasource-parquet/src/metadata.rs#L555-L585 - Row group pruning: https://github.com/apache/datafusion/blob/5a502dffce2f1881479e8c1a5b68a4a5108e7934/datafusion/datasource-parquet/src/row_group_filter.rs#L518-L531 - Page pruning: https://github.com/apache/datafusion/blob/5a502dffce2f1881479e8c1a5b68a4a5108e7934/datafusion/datasource-parquet/src/page_filter.rs#L760-L764 ### Describe the solution you'd like Trust `INT96` min/max statistics when column actually has `ColumnOrder::INT96_TIMESTAMP_ORDER`, and use them for file, row group, and page pruning. ### Describe alternatives you've considered N/A ### Additional context - Upgrade PR where the conservative check was added: https://github.com/apache/datafusion/pull/25335 - arrow-rs implementation: https://github.com/apache/arrow-rs/pull/10106 - Parquet format change: https://github.com/apache/parquet-format/pull/584 - Prior arrow-rs discussion of incorrect `INT96` statistics: https://github.com/apache/arrow-rs/issues/7686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
