adriangb opened a new pull request, #27005: URL: https://github.com/apache/datafusion/pull/27005
## Which issue does this PR close? - Related to apache/datafusion#26424 and apache/datafusion#26428. No issue of its own: this PR only adds benchmarks. ## Rationale for this change `MATERIALIZED` CTEs (apache/datafusion#26271) are planned differently from inlined CTEs: each body is optimized on its own before the rest of the plan. apache/datafusion#26428 adds more work to that step. No planner benchmark plans a materialized CTE today, so there is no standard way to see what this costs, or to compare it with inlining. ## What changes are included in this PR? New `sql_planner` cases, which plan a query from SQL to a physical plan. The PostgreSQL dialect is used, because sqlparser only parses `[NOT] MATERIALIZED` for that dialect. For each kind of CTE (unmarked, `MATERIALIZED`, `NOT MATERIALIZED`) there are three cases: - `physical_cte_<kind>_one_ref`: the CTE is read once. - `physical_cte_<kind>_two_refs`: the CTE is joined with itself. - `physical_cte_<kind>_chain_6`: six CTEs, each joining the previous one with itself. `physical_cte_mixed` has one CTE of each kind, each read twice. Results on `main` (Criterion means): | Case | unmarked | `MATERIALIZED` | `NOT MATERIALIZED` | | --- | --- | --- | --- | | one reference | 242 µs | 264 µs | 245 µs | | two references | 474 µs | 360 µs | 481 µs | | chain of 6 | 8.25 ms | 0.92 ms | 8.33 ms | `physical_cte_mixed` takes 1.36 ms. An inlined chain copies the first CTE 32 times. So it takes about 9 times as long to plan as the materialized chain. ## What is the testing strategy for this PR? This PR has only benchmarks. They run with: ```bash cargo bench -p datafusion --bench sql_planner -- physical_cte ``` The `sql_planner` benchmark needs ClickBench data. The checked-in fixture works: put `datafusion/core/tests/data/clickbench_hits_10.parquet` in a directory, and set `CLICKBENCH_DATA_PATH` to that directory. ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
