Gabriel39 opened a new pull request, #66412:
URL: https://github.com/apache/doris/pull/66412
## What problem does this PR solve?
File Scanner V2 kept multi-column `OR` predicates as residual expressions,
so TPC-DS Q28 could
not use dictionary-id filtering or raw direct filtering for its
price/coupon/cost disjunctions.
## What is changed?
- Flatten exact multi-column `OR` predicates into distinct single-column
branches.
- Build one compact truth map per branch with dictionary-id or raw
fixed/binary filtering, then OR
the maps before updating the selection vector.
- Reuse ordinary predicate readers for hidden single-use columns. Keep an
auxiliary reader only
when a projected column must later materialize the union survivors.
- Preserve SQL NULL semantics and use an all-or-nothing fallback for
unsupported expressions,
runtime filters, TopN filters, repeated columns, and unsafe expressions.
- Preflight every selected row group. The new policy is enabled only in
benchmark-validated
encoding/NULL-density ranges; otherwise the complete file keeps the
existing residual path.
- Add profile counters, Q28-shaped INT/DECIMAL tests, and a same-binary
DECIMAL(10,2)
microbenchmark matrix.
## Performance
The Release microbenchmark contains 100 legacy/raw pairs (200 registrations)
across PLAIN and
dictionary encoding, 0%/1%/10%/50%/90% NULL ratios, 1%/10%/50%/90%/100%
selectivity, and both
predicate-only and projected shapes. Runs were pinned to one CPU with warm
fixtures. Stable groups
used 7 repetitions at 0.2 seconds; noisy cases were repeated 10 times at 0.5
seconds.
Positive values mean the raw-disjunction path is faster.
| Encoding | NULL ratio | Shape | Median improvement | Minimum observed
improvement |
|---|---:|---|---:|---:|
| Dictionary | 10% | Predicate-only | 25.52% | 20.41% |
| Dictionary | 50% | Predicate-only | 77.13% | 62.19% |
| Dictionary | 50% | First predicate projected | 39.61% | 27.56% |
| Dictionary | 90% | Predicate-only | 24.34% | 14.76% |
| Dictionary | 90% | First predicate projected | 9.37% | 5.43% |
| PLAIN | 50% | Predicate-only | 14.30% | 3.32% |
| PLAIN | 90% | Predicate-only | 10.96% | 7.65% |
Across the 35 enabled pairs, the median improvement is 22.80% and every
stable result is positive.
All remaining matrix combinations are rejected by preflight and retain the
original residual scan
path, avoiding the regressions found by the initial unrestricted
implementation.
## Test
- `ninja -C be/ut_build_ASAN doris_be_test -j128`
- 92 ASAN tests:
`ParquetScanTest.*:ParquetBenchmarkScenariosTest.*:ParquetRuntimeFilterDirectReaderTest.*`
- `ninja -C be/build_Release benchmark_test -j128`
- 100 paired multi-column OR microbenchmark scenarios, plus targeted long
reruns for noisy cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
