Gabriel39 opened a new pull request, #65500:
URL: https://github.com/apache/doris/pull/65500
### What problem does this PR solve?
This PR consolidates and supersedes #65499, #65497, #65489, and #65493.
Parquet and ORC metadata pruning had several independent correctness gaps:
- missing Parquet null counts could incorrectly make nullable row groups
look null-free;
- UTC timestamp min/max conversion could become non-monotonic across DST
rollback;
- the Parquet bloom-filter cache was keyed only by column and could reuse a
filter across row groups;
- NaN min/max bounds could be treated as usable statistics and incorrectly
prune data.
The changes make missing null counts conservative, disable unsafe timestamp
range pruning across backward timezone transitions, scope bloom filters by row
group and column, and preserve null-count pruning while ignoring invalid NaN
bounds. ORC timestamp predicates use the same backward-transition safety check.
### Release note
Fix incorrect Parquet and ORC metadata pruning for missing null counts, DST
rollback, row-group bloom filters, and NaN statistics.
### Check List (For Author)
- Test: Unit Test
- ASAN_UT build and 12 targeted BE tests passed on `[email protected]`.
- Covered `ParquetStatisticsTransformTest.*`, the row-group bloom-cache
test, timestamp-statistics tests, and the ORC repeated-civil-time test.
- Behavior changed: Yes. Metadata pruning is now conservative for invalid or
non-monotonic statistics.
- Does this need documentation: No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
