alamb commented on code in PR #25335:
URL: https://github.com/apache/datafusion/pull/25335#discussion_r4050440530
##########
datafusion/datasource-parquet/src/metadata.rs:
##########
@@ -80,7 +80,12 @@ pub(crate) fn has_untrusted_min_max_order(
parquet_column_index: usize,
) -> bool {
let column = parquet_schema.column(parquet_column_index);
- if column.sort_order() == SortOrder::UNDEFINED {
+ // As of arrow 60, INT96 columns report `SortOrder::INT96_TIMESTAMP`
+ // rather than `UNDEFINED`; keep treating their min/max as untrusted.
+ if matches!(
+ column.sort_order(),
+ SortOrder::UNDEFINED | SortOrder::INT96_TIMESTAMP
Review Comment:
Yeah, sorry this was a confusing comment. I think we should eventually
support pruning INT96 timestamp columns, but we can't unless they have the
appropriate order annotation. I filed a ticket to explain this and and added a
comment to the code with a reference
- https://github.com/apache/datafusion/issues/25484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
