mapleFU commented on code in PR #34112:
URL: https://github.com/apache/arrow/pull/34112#discussion_r1104122046
##########
cpp/src/parquet/column_reader.cc:
##########
@@ -211,10 +211,15 @@ EncodedStatistics ExtractStatsFromHeader(const H& header)
{
return page_statistics;
}
const format::Statistics& stats = header.statistics;
- if (stats.__isset.max) {
+ // Use the new V2 min-max statistics over the former one if it is filled
+ if (stats.__isset.max_value && stats.__isset.min_value) {
+ // TODO: check if the column_order is TYPE_DEFINED_ORDER.
+ page_statistics.set_max(stats.max_value);
+ page_statistics.set_min(stats.min_value);
+ } else if (stats.__isset.max && stats.__isset.min) {
+ // TODO: check created_by to see if it is corrupted for some types.
Review Comment:
I've no problem here, but just curious, does this code meaning the
parquet-mr's `CorruptStatistics.shouldIgnoreStatistics`? (It's really
trickey...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
