etseidl opened a new pull request, #10106: URL: https://github.com/apache/arrow-rs/pull/10106
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #10105. - Depends on #10104 (which depends on #9619) # Rationale for this change Spark continues to use INT96 timestamps, despite INT96 being marked as deprecated in 2018. Query engines want valid statistics to allow reliably pruning on INT96 columns. https://github.com/apache/parquet-format/pull/584 adds a new `ColumnOrder` variant which can be used to signal compliance with the only known use of INT96 (4-byte julian day from epoch, 8-byte nanosecond). # What changes are included in this PR? Adds support for the new enum variant, and writes the appropriate value in the `FileMetaData.column_orders` field. This builds on changes introduced in #7687. # Are these changes tested? Yes # Are there any user-facing changes? Yes, this adds a new variant to public enums (`ColumnOrder::INT96_TIMESTAMP_ORDER`, `SortOrder::INT96_TIMESTAMP`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
