seawinde commented on code in PR #63899:
URL: https://github.com/apache/doris/pull/63899#discussion_r3717071383
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/PartitionIncrementMaintainer.java:
##########
@@ -803,6 +826,51 @@ public Plan getOriginalPlan() {
return originalPlan;
}
+ private Expression shuttleExpressionWithLineage(Expression expression,
Plan plan) {
+ return shuttleExpressionWithLineage(ImmutableList.of(expression),
plan).get(0);
+ }
+
+ private List<? extends Expression> shuttleExpressionWithLineage(List<?
extends Expression> expressions,
+ Plan plan) {
+ if (expressions.isEmpty()) {
+ return ImmutableList.of();
+ }
+ ExpressionLineageReplacer.ExpressionReplaceContext replaceContext =
+ new
ExpressionLineageReplacer.ExpressionReplaceContext(expressions);
+ for (NamedExpression namedExpression :
getLineageExpressionIndex(plan)) {
+ if
(!replaceContext.getUsedExprIdSet().contains(namedExpression.getExprId())) {
+ continue;
+ }
+
namedExpression.accept(ExpressionLineageReplacer.NamedExpressionCollector.INSTANCE,
replaceContext);
+ }
+ return replaceContext.getReplacedExpressions();
+ }
+
+ private List<? extends Expression>
shuttleAndNormalizeExpressionWithLineage(
+ Collection<? extends Expression> expressions, Plan plan) {
+ if (expressions.isEmpty()) {
+ return ImmutableList.of();
+ }
+ List<? extends Expression> shuttledExpressions =
+
shuttleExpressionWithLineage(ImmutableList.copyOf(expressions), plan);
+ List<Expression> normalizedExpressions = new
ArrayList<>(shuttledExpressions.size());
+ for (Expression expression : shuttledExpressions) {
+
normalizedExpressions.add(EXPRESSION_NORMALIZATION.rewrite(expression,
expressionRewriteContext));
+ }
+ return normalizedExpressions;
+ }
+
+ private List<NamedExpression> getLineageExpressionIndex(Plan plan) {
+ List<NamedExpression> lineageExpressionIndex =
planLineageExpressionIndexes.get(plan);
+ if (lineageExpressionIndex == null) {
+ List<NamedExpression> collectedIndex = new ArrayList<>();
+ plan.accept(LineageExpressionCollector.INSTANCE,
collectedIndex);
+ lineageExpressionIndex = collectedIndex;
+ planLineageExpressionIndexes.put(plan, lineageExpressionIndex);
Review Comment:
Thanks for the detailed analysis. We agree that overlapping nested subtrees
can retain a quadratic number of references in the worst case, and that the
current performance SQL does not cover a deeply projected join shape. This is a
performance and peak-memory trade-off rather than a lineage-correctness issue:
the cache is scoped to one `PartitionIncrementCheckContext` and retains
references rather than copying expressions. For this PR, we want to keep the
scope focused on the highest-impact hot path observed in the target workload.
We do not currently have profiling or a reproducer showing that this nested
projected-join shape causes material FE heap/GC pressure, so we prefer not to
add subtree-range, eviction, or additional index machinery here. We can address
it as a separate optimization if workload evidence shows a meaningful impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
