adriangb opened a new issue, #26239: URL: https://github.com/apache/datafusion/issues/26239
### Is your feature request related to a problem or challenge? Follow-up to https://github.com/apache/datafusion/pull/25580 (`WITH ... AS MATERIALIZED`), part of https://github.com/apache/datafusion/issues/17737. Related to the EPIC https://github.com/apache/datafusion/issues/22676 (`CteFilterPusher`). The body of a materialized CTE is shared by all references, so `PushDownFilter` moves a filter only onto the rest of the query and never into the body. The body is computed and buffered in full, even when each reference reads a small part of it. ```sql WITH c AS MATERIALIZED (SELECT * FROM events) SELECT * FROM c WHERE day = '2026-01-01' UNION ALL SELECT * FROM c WHERE day = '2026-01-02'; ``` All of `events` is read and buffered. With the CTE inlined, each branch reads one day. ### Describe the solution you'd like A logical optimizer rule that runs after filter pushdown has moved the filters next to the references: - If each reference to the CTE has a filter directly above it, push the disjunction `f1 OR f2 OR ...` into the body. - Keep the original filter on each reference, because the body now returns the rows for all references. - If one reference has no filter, do nothing. In the example, the body gets `day = '2026-01-01' OR day = '2026-01-02'`. DuckDB has a rule of this kind. Only deterministic filters that refer to columns of the CTE can be used. The rule must not change the result of a volatile body. ### Describe alternatives you've considered Inline the CTE when the filters are selective. That is part of https://github.com/apache/datafusion/issues/26237. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
