adriangb opened a new pull request, #26274:
URL: https://github.com/apache/datafusion/pull/26274
## Which issue does this PR close?
- Closes apache/datafusion#26240.
Stacked on apache/datafusion#25580 and on #26269 (apache/datafusion#26238).
Only the last commit is new.
## Rationale for this change
The body of a materialized CTE kept all its columns, also when no reference
read them.
```sql
CREATE TABLE wide(a INT, b INT, c INT, d INT, e INT) AS VALUES (1, 10, 100,
1000, 10000);
EXPLAIN WITH w AS MATERIALIZED (SELECT * FROM wide)
SELECT a FROM w
UNION ALL
SELECT b FROM w;
```
Before, all five columns were read and buffered:
```text
MaterializedCte: name=w
SubqueryAlias: w
TableScan: wide projection=[a, b, c, d, e]
Union
Projection: w.a
MaterializedCteScan: name=w
Projection: w.b AS a
MaterializedCteScan: name=w
```
After, only `a` and `b` are read and buffered:
```text
MaterializedCte: name=w
SubqueryAlias: w
TableScan: wide projection=[a, b]
Union
MaterializedCteScan: name=w projection=[a]
Projection: w.b AS a
MaterializedCteScan: name=w projection=[b]
```
## What changes are included in this PR?
- `MaterializedCteScan` has a new field `projection: Option<Vec<usize>>`:
the indexes of the columns of the CTE body that the scan outputs.
- `OptimizeProjections` narrows each `MaterializedCteScan` to the columns
that its parent needs, as it does for a `TableScan`.
- At a `MaterializedCte`, `OptimizeProjections` optimizes the continuation
first. It then computes the union of the columns of all the scans of the CTE,
projects the body to this union, and gives each scan the new positions of its
columns. Scans in subquery expressions and in the bodies of other CTEs are
counted.
- `MaterializedCteScanExec` applies the projection to the shared buffered
batches.
- The physical planner checks that each scan agrees with the physical schema
of its body when it binds the scan, and returns an internal error if it does
not.
The projection in the scan makes the intermediate state (a narrow scan and a
wide body) a valid plan, so no separate rule is necessary.
Notes:
- A reference that reads no column (`SELECT count(*) FROM c`) gets
`projection=[]`. If no reference reads a column, the body keeps its first
column.
- The projection is put above the body. A volatile column can be removed,
but the number of rows does not change. Columns below a `DISTINCT` or a
recursive query are not removed.
- A scan in a correlated subquery that is not decorrelated is not narrowed.
The CTE then keeps all its columns.
- The pgjson display does not show the projection.
## Are these changes tested?
Yes. `cte_materialized.slt`: `EXPLAIN` and result checks for the example
above, a column that only a filter reads, `count(*)`, a CTE that another CTE
reads, references in subqueries, a volatile body, a `DISTINCT` body, aliased
columns, a self join, a recursive body and a CTE that nothing reads. Unit tests
for the scan projection, nested CTEs, idempotence and the zero-column case.
## Are there any user-facing changes?
`EXPLAIN` shows `projection=[...]` on `MaterializedCteScan` and
`MaterializedCteScanExec` when the scan does not read all the columns.
`MaterializedCteScan` has the new public field `projection`. The upgrade guide
for 56.0.0 says this.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
