Gabriel39 opened a new pull request, #70005:
URL: https://github.com/apache/doris/pull/70005
### What problem does this PR solve?
Issue Number: None
Related PRs: #69655, #69946
Backport #69655 to `branch-4.2`. FLOAT/DOUBLE min/max bounds omit NaNs, so
they cannot independently prove that comparisons or `NOT IN` exclude a row
group or page. An explicit, validated `nan_count=0` now clears the existing
NaN-unknown guard for the exact owning column chunk. Missing, positive,
invalid, or inconsistent metadata keeps the conservative fallback.
The backport retains branch-4.2's statistics conversion API and direct
predicate traversal. It adds standard optional Parquet Statistics field 9,
matching master; existing field meanings remain consistent across branches 3.1,
4.0, 4.1, 4.2, and master.
**Applicability of #69946:** That PR fixes `PluginDrivenExternalTable`,
which exists on master and is absent from branch-4.2. The existing
`IcebergExternalTable.loadSnapshot()` on branch-4.2 already checks `isView()`
and returns `EmptyMvccSnapshot` before requesting table snapshot metadata. Its
relevant behavior is already present, so this PR contains no FE changes for
#69946.
### Release note
Improve Parquet floating-point min/max pruning when trustworthy zero NaN
counts are available in file metadata.
### Check List (For Author)
- Test:
- [x] Focused ASAN compilation: modified production and unit-test
translation units, plus generated Parquet Thrift code.
- [ ] Unit-test execution: not run locally; complete BE test execution is
deferred to CI. Backported tests cover footer/page pruning, FLOAT/DOUBLE,
NULL/NaN preservation, invalid counts, and column/row-group proof scope.
- [x] Manual validation: clang-format 16 on all four affected C++ files,
cross-branch Thrift field compatibility, complete diff review,
sensitive-content scan, and `git diff --check`.
- Behavior changed:
- [x] Yes. Proven NaN-free floating chunks and pages can be skipped by
existing comparison and `NOT IN` predicates.
- Does this need documentation?
- [x] No. Internal reader optimization using existing file metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
