github-actions[bot] commented on code in PR #66898:
URL: https://github.com/apache/doris/pull/66898#discussion_r4357564780
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/RewriteCteChildren.java:
##########
@@ -315,4 +338,81 @@ private LogicalPlan tryToConstructFilter(CascadesContext
cascadesContext, CTEId
}
return child;
}
+
+ /**
+ * Collects the demand that the plan above {@code cteAnchor} places on the
columns of the CTE,
+ * resolved across the retained CTEs it depends on.
+ *
+ * <p>A demand is stated on the outputs of the consumer of this CTE, but a
retained CTE may feed
+ * another one, e.g.
+ * <pre>
+ * WITH c1 AS (SELECT a FROM wide),
+ * c2 AS (SELECT a FROM c1 UNION ALL SELECT a FROM c1)
+ * SELECT a FROM c2 INTERSECT SELECT a FROM c2
+ * </pre>
+ * where the set operation demands {@code c2}, whose producer reads {@code
c1}. A consumer copies
+ * the slots of the producer it reads and {@link OperativeColumnDerive}
translates a demand on a
+ * consumer output to the slot of that producer, so one walk resolves one
hop of the chain; the
+ * walk is repeated with the ids found so far until no new slot is
demanded, which lets the
+ * demand reach the producer of this anchor before its statistics are
derived.
+ */
+ static OperativeColumnDerive.DeriveContext deriveConsumerDemand(
+ LogicalCTEAnchor<?, ?> cteAnchor, CascadesContext cascadesContext)
{
+ RoaringBitmap demand = new RoaringBitmap();
+ int hops = 1 + cteAnchor.child(1)
+ .<LogicalCTEAnchor<?,
?>>collectToList(LogicalCTEAnchor.class::isInstance).size();
+ for (int hop = 0; hop < hops; hop++) {
+ OperativeColumnDerive.DeriveContext context = new
OperativeColumnDerive.DeriveContext();
+ context.operativeSlotIds.or(demand);
+ cteAnchor.child(1).accept(new OperativeColumnDerive(), context);
Review Comment:
[P2] Share demand propagation across retained CTE anchors
Extend the retained-chain case to n CTEs, each reading its predecessor twice
with UNION ALL, followed by INTERSECT over two consumers of the last CTE. These
anchors remain materialized with the default reference threshold. The reduced
tree is:
```text
Anchor(c1: Scan(a))
Anchor(c2: UnionAll(Consumer(c1), Consumer(c1)))
...
Intersect(Consumer(cn), Consumer(cn))
```
OperativeColumnDerive visits each producer before the final consumer query,
so one full walk exposes only one earlier dependency. This loop therefore walks
the entire remaining suffix O(n) times; visitLogicalCTEAnchor then repeats that
process independently at every subsequent anchor. Demand collection becomes
O(n^3), even though the retained plan has O(n) nodes. For example, 200 one-key
CTEs require at least 13,393,300 visitor calls and 8,019,900 visitChildren
builder constructions in this helper alone; wider outputs multiply the per-node
work. The rewritten trees are discarded, and the ordinary before/after rewrite
phases repeat the collection.
Please collect the demand/dependency graph once and propagate it in reverse
dependency order or with a shared worklist, then reuse the result when deriving
each producer. Add coverage that bounds traversal work for a longer retained
chain. This is distinct from the existing missing-transitive-demand thread:
that omission is repaired, but the repair introduces this repeated work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
