adriangb opened a new pull request, #26269:
URL: https://github.com/apache/datafusion/pull/26269
## Which issue does this PR close?
- Closes apache/datafusion#26238.
Stacked on apache/datafusion#25580. Only the last commit is new.
## Rationale for this change
apache/datafusion#25580 plans `WITH x AS MATERIALIZED (...)` with two
built-in `UserDefinedLogicalNode`s inside `LogicalPlan::Extension`. Rules that
must know about the node find it with a downcast, and code outside DataFusion
sees an unknown extension node.
For users of SQL nothing changes. Plans and results are the same. For
library users, the nodes are now visible in a `match`:
```rust
// before
LogicalPlan::Extension(Extension { node }) if
node.as_any().is::<MaterializedCte>() => { /* downcast */ }
// after
LogicalPlan::MaterializedCte(cte) => { /* cte.body(), cte.continuation() */ }
LogicalPlan::MaterializedCteScan(scan) => { /* scan.id() */ }
```
It is easier to do this before more rules are written against the extension
node.
## What changes are included in this PR?
- New variants `LogicalPlan::MaterializedCte` and
`LogicalPlan::MaterializedCteScan`, in the style of
`LogicalPlan::RecursiveQuery`.
- The reference is a new leaf variant and not a `TableScan` over a
`TableSource`. A `TableScan` would need a downcast of the source to read the
CTE id, and `OptimizeProjections` would write a projection into it, which
changes the plans.
- `MaterializedCte::try_new` checks that each scan of the CTE in the
continuation has the column names and types of the body. `with_new_exprs` and
the `TreeNode` rewrites do not run this check again.
- `MaterializedCteScan::find_in` returns the scans of one CTE in a plan,
also in subquery expressions.
- The downcasts are removed from `PushDownFilter`, `PullUpCorrelatedExpr`,
`LogicalPlan::head_output_expr` and the physical planner. Each `match` now
states the behaviour, and the behaviour is the same as before.
- The unparser, the Substrait producer and the protobuf serializer return a
"not implemented" error for the two variants. Protobuf support is
apache/datafusion#26231.
## Are these changes tested?
Yes. `cte_materialized.slt` passes without changes, so no plan changed.
There are new unit tests for the constructor check and for `find_in`. The "not
implemented" arms have no tests.
## Are there any user-facing changes?
Yes, this is a public API change. An exhaustive `match` on `LogicalPlan` in
downstream code needs two new arms. `MaterializedCte::cte` and
`MaterializedCte::continuation` are now `Arc<LogicalPlan>`. There is an entry
in the 56.0.0 upgrade guide.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
