adriangb opened a new pull request, #25453: URL: https://github.com/apache/datafusion/pull/25453
## Which issue does this PR close? <!-- No issue: this PR adds test coverage only. --> - Related to https://github.com/apache/datafusion/issues/25414, https://github.com/apache/datafusion/pull/25412 and https://github.com/apache/datafusion/issues/25268. ## Rationale for this change `ExtractLeafExpressions` and `PushDownLeafProjections` (the option `datafusion.optimizer.enable_leaf_expression_pushdown`, default `true`) move expressions such as `s['b']` out of a filter, sort, limit, aggregate or join, and push them towards the scan. The rules resolve columns by name. When a plan holds two columns with the same name, or a column that is computed from a column of the same name, the rules can drop a projection or merge the wrong pair. The user sees wrong rows or a planning error, with the default configuration. The rules had 10 bug fixes since February 2026. Users or an ad-hoc fuzzer found all of them. The 520 sqllogictest files found none, because they do not build these shapes. A differential fuzz test finds this class cheaply: run the same query with the option on and off, and compare the rows. Bugs of the class this test covers: - https://github.com/apache/datafusion/issues/25414: a computed column with the same name as its input is dropped. Wrong results. - https://github.com/apache/datafusion/pull/25412: pass-through columns merged in the wrong name space. Planning error. - https://github.com/apache/datafusion/issues/24678: a volatile expression duplicated by leaf extraction. Wrong results. - https://github.com/apache/datafusion/issues/25415: a filter on a volatile group key pushed below the aggregate. Wrong results. - https://github.com/apache/datafusion/issues/22895: `optimize_projections` fails with "No field named ..." when the join keys hold a `get_field`. Planning error. - https://github.com/apache/datafusion/issues/24241: `push_down_leaf_projections` fails for a self join. Planning error. The harness reproduces the two open bugs today. With `INCLUDE_SAME_NAME_ALIAS_SHAPES = true`, 526 of 3000 seeds fail (476 planning errors, 50 silent wrong results). With `INCLUDE_FALSE_BRANCH_UNION_SHAPES = true`, 404 of 3000 seeds fail. ## What changes are included in this PR? One new file, `datafusion/core/tests/fuzz_cases/leaf_pushdown_fuzz.rs`. The generator builds SQL over three wide tables and one narrow table. Every table has `Int32` columns, a `Utf8` column and two `Struct` columns. Two struct fields have the same name as a top level column. The generated shapes are: - `s['f']` in `SELECT`, `WHERE`, `ORDER BY`, `GROUP BY`, `HAVING`, `JOIN ON` and under `LIMIT`. - Derived tables, CTEs, `IN` and `EXISTS` subqueries, `UNION ALL`. - `named_struct(...)` built in a subquery and read above. - `arrow_field(a)` used bare and through `['f']` in the same query. - `random()` in a subquery projection, read two times above. The two runs cannot be compared with each other, so the query reports its own consistency in one boolean column. Each query runs in two `SessionContext`s built from the same batches. The rows are sorted and compared. An error on one side only is a failure. An error on both sides is a skip, and the test fails if more than 30% of the seeds skip. Three shapes fail on `main` today. Each one is behind a named `const` switch with the link to its issue, so this PR is green: - `INCLUDE_SAME_NAME_ALIAS_SHAPES` for https://github.com/apache/datafusion/issues/25414. - `INCLUDE_FALSE_BRANCH_UNION_SHAPES` for https://github.com/apache/datafusion/pull/25412. Flip the switch to `true` when the fix lands. ## What is the testing strategy for this PR? This PR is a test. Three test entry points: | test | cases | wall clock | | --- | --- | --- | | `leaf_pushdown_fuzz` | 250 | 3.9s | | `leaf_pushdown_parquet_fuzz` | 125 | included above | | `leaf_pushdown_parquet_schema_evolution` | 50 | included above | | `leaf_pushdown_fuzz_extended` | 5000 + 1000 | 87s | ```console cargo test --profile ci -p datafusion --features extended_tests --test fuzz -- leaf_pushdown ``` The whole `fuzz_cases` tree is already behind the `extended_tests` feature, so the short tests run in the extended job together with the long one. Every case comes from a seed. A failure prints the seed, the SQL and both result sets, and names the command that replays it: ```console LEAF_PUSHDOWN_FUZZ_SEED=<seed> LEAF_PUSHDOWN_FUZZ_CASES=1 \ cargo test --profile ci -p datafusion --features extended_tests \ --test fuzz -- leaf_pushdown --nocapture ``` ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
