divjotarora opened a new pull request, #584:
URL: https://github.com/apache/parquet-format/pull/584
### Rationale for this change
When writing INT96 timestamp columns, writers either omit stats altogether
or emit stats using the `TYPE_ORDER` ordering. However, some writers were
incorrectly emitting stats via bytewise comparisons, which does not result in
chronological INT96 ordering. These stats are incorrect and must be ignored by
readers. The goal of this change is to introduce a mechanism for readers to
correctly determine the validity of an INT96 column's statistics and ignore
them if they are potentially incorrect.
### What changes are included in this PR?
This PR specifies a new `INT96_TIMESTAMP_ORDER` sort order specifically used
for INT96 timestamp statistics. Additionally, it suggests writers use this
ordering when emitting INT96 stats and that readers *ignore* stats for INT96
`TYPE_ORDER`'d columns.
### Do these changes have PoC implementations?
In-progress
<!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the
actual Github issue id. -->
Closes #$583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
