adriangb opened a new pull request, #20117: URL: https://github.com/apache/datafusion/pull/20117
## Summary This PR adds a new optimizer rule `ExtractLeafExpressions` that extracts `MoveTowardsLeafNodes` sub-expressions (like `get_field`) from Filter, Sort, Limit, Aggregate, and **Projection** nodes into intermediate projections. This normalization allows `OptimizeProjections` (which runs next) to merge consecutive projections and push `get_field` expressions down to the scan, enabling Parquet column pruning for struct fields. ### Example ```sql SELECT id, s['label'] FROM t WHERE s['value'] > 150 ``` **Before:** `get_field(s, 'label')` stayed in ProjectionExec, reading full struct `s` **After:** Both `get_field` expressions pushed to DataSourceExec: ``` DataSourceExec: projection=[get_field(s, value) as __leaf_5, get_field(s, label) as __leaf_4, id] ``` ### How It Works The rule: 1. Extracts `MoveTowardsLeafNodes` expressions into `__leaf_N` aliases 2. Creates inner projections with extracted expressions + pass-through columns 3. Creates outer projections to restore original schema names 4. Handles deduplication of identical expressions 5. Skips expressions already aliased with `__leaf_*` to ensure idempotency ### Interaction with OptimizeProjections **Optimizer order** (from `optimizer.rs`): ```rust Arc::new(ExtractLeafExpressions::new()), // Runs FIRST Arc::new(OptimizeProjections::new()), // Runs SECOND - merges projections ``` ## Test plan - [x] New unit tests for projection extraction in `extract_leaf_expressions.rs` - [x] Updated sqllogictest expectations in `projection_pushdown.slt` - [x] All optimizer tests pass (`cargo test -p datafusion-optimizer`) 🤖 Generated with [Claude Code](https://claude.ai/code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
