Gabriel39 opened a new pull request, #69655: URL: https://github.com/apache/doris/pull/69655
### What problem does this PR solve? Problem Summary: Parquet V2 treats every FLOAT/DOUBLE column as having an unknown NaN count. Finite min/max bounds omit NaNs, so the existing comparison and NOT IN guards correctly retain groups that might contain a matching NaN. This also prevents pruning when a writer explicitly supplies `nan_count=0`. For example, a column chunk with `min=max=1` and a trustworthy zero NaN count cannot match `x > 1`, `x >= 2`, `x <> 1`, or `x NOT IN (1)`. Previously the reader decoded those rows anyway. This change makes that zero proof available to the existing ZoneMap evaluator. ### Optimization approach - Add standard optional Statistics field 9, `nan_count`, matching the [Apache Parquet format definition](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift). Compared the touched definition on master and branch-3.1/4.0/4.1/4.2: existing field IDs retain their meanings and field 9 is unused on those Doris branches. Older readers skip the optional field, and existing files retain the conservative fallback. - Retain a zero-count proof in converted FLOAT/DOUBLE statistics only when the count is explicitly zero, the value/null counts are consistent, and the decoded bounds are usable (or the chunk is all NULL). Missing, positive, negative, or excessive counts do not clear the NaN-unknown flag. Invalid/NaN/unordered bounds remain unusable. - Pass the proof alongside the row-group ZoneMap. For PageIndex evaluation, reuse only the proof from the exact owning row group and physical leaf column. It covers every page of that chunk. Both single-slot and compound OR/AND page evaluation use this scoped proof. - Keep the existing comparator and NOT IN semantics. Integer/string columns and legacy floating files avoid the additional footer decode in the page proof path. Writer support and additional column orders are outside this reader change. ### Optimization effect | Per reader query | Absent count (previous fallback) | Explicit zero count | | --- | ---: | ---: | | RawRowsRead | 524,288 | 4,096 (-99.22%) | | RowGroupsFilteredByMinMax | 0 | 127 | | Actual local read bytes | 6,358,496 | 98,444 (-98.45%) | | Median wall time | 6.908 ms | 0.690 ms (~10.0x) | | Median process CPU time | 6.868 ms | 0.687 ms (~10.0x) | Measured using Clang 20 RELEASE, warm local filesystem cache, one pinned CPU. The absent-count samples range from 6.787 to 7.252 ms and zero-count samples from 0.673 to 0.730 ms. All returned IDs and per-query work counters match their expectations across all samples. The opt-in `ParquetScanTest.ZeroNanCountReaderMicrobench` uses the actual V2 reader and corresponds to: ```sql SELECT id FROM parquet_source WHERE d > 1; ``` Fixture: 128 row groups x 4,096 rows; the first 127 groups contain `d=1`, the last contains `d=2`. FLOAT, DOUBLE, and an integer row identifier are stored using uncompressed PLAIN DataPageV2 with dictionary and PageIndex disabled. Two files have identical data pages and differ only in the optional zero-count fields. The absent-count variant exercises the same conservative behavior as the pre-change reader. Each query checks every returned ID and the pruning counters. Seven samples of eight reader queries per variant follow warmup, alternating variant order. The benchmark records wall time, process CPU time, actual local read bytes, RawRowsRead, and RowGroupsFilteredByMinMax. Results describe a local reader microbenchmark; full SQL timings depend on storage, parallelism, and the rest of the query. ### Validation and reproduction - Related BE suites: 372 tests passed; the opt-in benchmark was skipped in that run and then passed separately. - The three initial reader regressions first failed on unmodified master because zero counts did not reduce RawRowsRead or increase pruning. Their result rows were already correct; the missing behavior was pruning. - Generated C++ Thrift code compiled successfully with the optional field. Old files without the field remain covered by the complete reader suite. - clang-format 16 and BE build-hygiene checks passed. Changed-line clang-tidy passed on all four affected C++ files. Coverage includes FLOAT/DOUBLE; zero/positive/absent/negative/excessive counts; GT/GE/NE/NOT IN; invalid null counts and bounds; all NULL/all NaN/mixed NULL-finite-NaN rows; dictionary on/off; DataPageV1/V2; PageIndex on/off; column/row-group proof scope; and compound OR page pruning when one branch lacks a zero proof. Mixed NaN cases are also checked against a statistics-disabled file. Correctness suites: ```bash BUILD_TYPE_UT=RELEASE ./run-be-ut.sh -j8 --run \ --filter='*Parquet*:*ColumnMapper*:*TableReader*' ``` Opt-in benchmark: ```bash DORIS_PARQUET_NAN_COUNT_BENCHMARK=1 BUILD_TYPE_UT=RELEASE \ ./run-be-ut.sh -j8 --run --filter=ParquetScanTest.ZeroNanCountReaderMicrobench ``` ### Release note Improve Parquet V2 floating-point Min-Max pruning when trustworthy zero NaN counts are available. ### Check List (For Author) - Test - [x] Unit Test - [x] Manual test (reader microbenchmark) - Behavior changed: - [x] Yes. Proven NaN-free floating chunks/pages can be skipped by existing comparison and NOT IN predicates. - Does this need documentation? - [x] No. Internal reader optimization using existing file metadata. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
