xudong963 opened a new pull request, #26404:
URL: https://github.com/apache/datafusion/pull/26404
## Which issue does this PR close?
- Closes #26349.
## Rationale for this change
`AggregateStatistics` can answer a scalar subquery such as `(SELECT min(ts)
FROM live)` from statistics and replace it with a row of literals. A join
against that row still runs as a `NestedLoopJoinExec`.
That join only filters its other input, but `NestedLoopJoinExec` preserves
neither input's ordering. Operators above it therefore lose the input's
ordering. For example, an aggregation grouped by the input's sort key can no
longer use `ordering_mode=Sorted` and must hold every group until the end. The
literal bound is also not pushed into the scan for pruning.
## What changes are included in this PR?
A new physical optimizer rule, `EliminateConstantJoin`, runs right after
`AggregateStatistics`. It applies when one input of an inner
`NestedLoopJoinExec` is provably a single row of literals: projections over a
single-partition `PlaceholderRowExec`. The rule substitutes the literals into
the join filter. It then replaces the join with a `FilterExec` over the other
input, under a `ProjectionExec` that restores the join's output schema and
metadata.
At this point in the optimizer, a folded `min(ts) AS first_ts` is still an
alias projection over the literal projection, so the rule follows stacked
projections.
The rule leaves the join unchanged when:
- the join is not an inner join;
- the placeholder has more than one partition, because it emits one row per
partition;
- a value is not a literal; for example, `random()` would then be evaluated
per row instead of once;
- the join filter contains a lambda, because lambda variables are numbered
after the filter's own columns, not the other input's.
## What is the testing strategy for this PR?
- `eliminate_constant_join.slt` has plans and results for two rewrites: a
scalar folded from Parquet statistics, and a literal subquery. In the first,
the aggregation becomes `ordering_mode=Sorted` and the bound is pushed into the
Parquet scan. The file also checks that a left join and a `random()` constant
keep the join.
- `physical_optimizer/eliminate_constant_join.rs` covers cases SQL cannot
reach:
- a multi-partition placeholder;
- a lambda join filter, planned without optimizer rules because projection
pushdown cannot plan it yet (#26399);
- schema metadata that comes from the constant input.
- `explain.slt`: `EXPLAIN VERBOSE` lists the new rule.
## Are there any user-facing changes?
Queries that join against a row folded from statistics, or against a literal
subquery, now plan a filter and a projection instead of a nested loop join.
`EXPLAIN VERBOSE` and `PhysicalOptimizer::new()` include the new rule. There
are no API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
