github-actions[bot] commented on code in PR #67152:
URL: https://github.com/apache/doris/pull/67152#discussion_r3871021994
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FillUpQualifyMissingSlot.java:
##########
@@ -189,10 +287,63 @@ interface PlanGenerator {
Plan apply(Set<Expression> newConjuncts, List<NamedExpression>
projects);
}
- private Plan createPlan(LogicalProject<Plan> project, Set<Expression>
conjuncts, PlanGenerator planGenerator) {
+ private Plan createPlan(LogicalProject<Plan> project, Set<Expression>
conjuncts,
+ Optional<Scope> outerScope, PlanGenerator planGenerator) {
Set<Slot> projectOutputSet = project.getOutputSet();
List<NamedExpression> newOutputSlots = Lists.newArrayList();
Set<Expression> newConjuncts = new LinkedHashSet<>();
+
+ // A correlated column referenced in qualify may be hidden behind a
project alias, e.g.
+ // `QUALIFY f = 1` where f is aliased as an outer column o.flag. If
the project also
+ // contains a window expression, filter pushdown cannot rewrite f back
to its producer
+ // before apply decorrelation, so the alias-producer dependency would
be lost and the
+ // correlation slot would never be collected into the apply. Resolve
such aliases whose
+ // producers reference only outer correlated slots, so the correlation
stays visible to
+ // subquery unnesting.
+ Map<Slot, Expression> correlatedAliasToProducer = Maps.newHashMap();
+ // If any qualify conjunct contains a subquery (e.g. an IN/NOT IN
predicate), the alias
+ // replacement below would descend into the subquery and break the
apply decorrelation
+ // (the substituted slot would be owned by neither side of the apply).
Skip the rewrite
+ // in that case.
+ boolean conjunctsHaveSubquery = conjuncts.stream().anyMatch(c ->
c.containsType(SubqueryExpr.class));
Review Comment:
[P1] Scope this fence to the subquery-bearing conjunct
This query-wide flag also disables safe alias repair in unrelated conjuncts.
For example, the right side of an outer EXISTS can contain:
```text
Qualify[rn = 1, f = 1, scalarSubquery(j) > 0]
Project[i.k, o.flag AS f, row_number(...) AS rn]
Scan(i)
```
Because only the scalar conjunct contains a `SubqueryExpr`, replacing `f ->
o.flag` in the separate `f = 1` conjunct would not cross that subquery's Apply
boundary. This global check skips the map entirely, however, so the enclosing
Apply never gets a correlation filter for `o.flag`; `ExistsApplyToJoin` takes
its uncorrelated path while the right project still consumes the left-only
slot, and final slot validation fails. Rewrite only subquery-free conjuncts
while retaining the fence for the unsafe conjunct itself, and add a
full-pipeline mixed scalar/EXISTS regression.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/FillUpQualifyMissingSlot.java:
##########
@@ -158,23 +222,57 @@ public List<Rule> buildRules() {
qualify -> having -> project(distinct)
*/
RuleType.FILL_UP_QUALIFY_HAVING_PROJECT.build(
- logicalQualify(logicalHaving(logicalProject())).then(qualify
-> {
+ logicalQualify(logicalHaving(logicalProject())).thenApply(ctx
-> {
+ LogicalQualify<LogicalHaving<LogicalProject<Plan>>>
qualify = ctx.root;
checkWindow(qualify);
+ Optional<Scope> outerScope =
ctx.cascadesContext.getOuterScope();
LogicalHaving<LogicalProject<Plan>> having =
qualify.child();
LogicalProject<Plan> project = qualify.child().child();
- return createPlan(project, qualify.getConjuncts(),
(newConjuncts, projects) -> {
+ return createPlan(project, qualify.getConjuncts(),
outerScope, (newConjuncts, projects) -> {
ImmutableList<NamedExpression> copyOutput =
ImmutableList.copyOf(project.getOutput());
if (project.isDistinct()) {
+ // Keep correlated predicates that only depend on
outer slots together with the
+ // having's own correlated predicates, on the same
decorrelatable side of the
+ // distinct barrier, so subquery unnesting can
collect and decorrelate them
+ // together (otherwise one of them is left
dangling in the apply's right side).
+ // A predicate that is constant per outer row is
equivalent before/after
+ // distinct, so moving it above the distinct
project preserves semantics.
+ Set<Expression> newHavingConjuncts = new
LinkedHashSet<>(having.getConjuncts());
+ Set<Expression> distinctQualifyConjuncts = new
LinkedHashSet<>();
+ if (outerScope.isPresent()) {
+ Set<Slot> correlatedSlots =
outerScope.get().getCorrelatedSlots();
+ for (Expression conjunct : newConjuncts) {
+ Set<Slot> inputSlots =
conjunct.getInputSlots();
+ // Only relocate deterministic predicates:
visible outer slots do
+ // not make a predicate constant (e.g.
`o.flag <> 1 OR random() < 0.5`
+ // has only {o.flag} as input slots), and
moving a volatile
+ // predicate changes its evaluation domain
(per row before DISTINCT
+ // vs once after DISTINCT), so keep
volatile predicates on their
+ // original side of the distinct barrier.
+ if (!inputSlots.isEmpty() &&
correlatedSlots.containsAll(inputSlots)
Review Comment:
[P1] Preserve nested subquery dependencies across DISTINCT
`getInputSlots()` deliberately excludes a subquery's inner plan, so this
containment test can classify a row-dependent subquery predicate as outer-only.
A reduced case is:
```text
Qualify[rn = 1, o.flag = 1 OR EXISTS(j.v = i.not_grouped)]
ProjectDistinct[i.k]
Scan(i.k, i.not_grouped)
```
The conjunct reports only `{o.flag}` here and is moved into HAVING.
Bottom-up `SubqueryToApply` must then create the nested EXISTS Apply above
`ProjectDistinct[i.k]`, where `i.not_grouped` has no owner, so final validation
rejects the plan. Merely fencing the motion is not sufficient for this
non-output correlation because the current missing-slot logic also cannot see
the hidden `i.not_grouped` dependency. Surface the complete nested correlation
in the pre-DISTINCT project while keeping the predicate in its original
evaluation domain, or reject this combined shape stably, and add nested
scalar/EXISTS coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
