Baymine opened a new pull request, #66109:
URL: https://github.com/apache/doris/pull/66109
### What problem does this PR solve?
Issue Number: close #66108
Problem Summary:
The ORC and Parquet readers evaluate every filter conjunct over all rows of a
block and AND the results together, regardless of how cheap/selective each
predicate is or the order the planner emitted them. An expensive predicate
(e.g. a string function like split_by_string(col)[n]='x') is therefore run
over
the full block even when a cheap, highly-selective predicate on the same
block
would have filtered almost everything first.
This adds three session-gated, correctness-neutral optimizations that only
change the order/method of filter execution, never which rows pass:
1. Static cost-based conjunct reorder (enable_scan_conjunct_reorder):
VExpr::compute_conjunct_cost scores each conjunct by a structural per-node
cost (leaves/comparisons cheap, function calls expensive, summed over the
tree with constant subtrees skipped), and VExpr::reorder_conjuncts_by_cost
stable-sorts conjuncts cheap-first.
2. Selection-vector selective execution (enable_scan_selective_filter):
VExprContext::execute_conjuncts_selective threads the set of surviving row
indices between conjuncts, so a later (typically expensive) predicate is
evaluated only on rows earlier ones let through. It produces the identical
full-width result_filter / can_filter_all contract as execute_conjuncts,
so
it is a drop-in. Runtime-filter-wrapper conjuncts stay full-width (they
carry
selectivity/counter side effects) and the selective gather is skipped
while
the surviving fraction is still high (> 0.5), where gather is not worth
its
memcpy.
3. Adaptive runtime reorder (enable_scan_adaptive_reorder):
A per-VExprContext FilterRuntimeStats samples ~1/8 of batches (min 4096
timed
rows) and re-sorts conjuncts by measured (per-row cost / drop fraction),
correcting the static estimate for skewed data or unknown UDF cost.
All three default OFF (opt-in): with the flags unset the ORC/Parquet filter
path is byte-identical to before, so this carries zero risk for existing
queries and lets operators enable/benchmark it per session. A wrong
cost/selectivity estimate can only cost a suboptimal execution order, never a
wrong result.
### Release note
Add three session variables (all default false / opt-in, execution-order
only,
never change query results): enable_scan_conjunct_reorder,
enable_scan_selective_filter, enable_scan_adaptive_reorder — controlling
selectivity-aware filter execution in the ORC/Parquet readers.
### Check List (For Author)
- Test:
- Unit Test: Added be/test/exprs/vexpr_conjunct_reorder_test.cpp (10
cases:
cost model + static reorder) and
be/test/exprs/vexpr_selective_conjuncts_test.cpp
(20 cases: selective-vs-full-width equivalence incl. a 500-iter
randomized
fuzz, plus adaptive-reorder gating) — 30 BE tests pass. Extended
SessionVariablesTest for the three new vars (FE) — asserts default
false,
setter/toThrift round-trip, and forward-variable propagation.
- The four reader call sites (ORC _get_next_block_impl +
OrcReader::filter,
Parquet next_batch + _do_lazy_read) wrap the existing execute_conjuncts
with the session-gated reorder/selective path; with the flags off the
path
is byte-identical to before. No dedicated reader regression suite was
added
(requires a live cluster) — noted as follow-up.
- Behavior changed: No (opt-in, default off; execution order/method only when
enabled; results are identical either way).
- Does this need documentation: No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
