alamb opened a new pull request, #23821: URL: https://github.com/apache/datafusion/pull/23821
## Which issue does this PR close? This PR adds test coverage rather than closing an issue. It documents the current behavior of these bugs so that fixing them shows up as a test change: - https://github.com/apache/datafusion/issues/23634 - https://github.com/apache/datafusion/issues/23818 - https://github.com/apache/datafusion/issues/23819 - https://github.com/apache/datafusion/issues/23820 ## Rationale for this change DataFusion derives functional dependencies from `PRIMARY KEY` / `UNIQUE` constraints and from `GROUP BY` keys, and several optimizer rules consume them to remove redundant work. Coverage today is scattered through `group_by.slt`, which makes it hard to tell which behaviors are intentional and which are bugs. While reviewing #23636 it was not obvious which of the surrounding wrong answers were pre-existing and which the PR introduced. This file pins down the starting point: it is written against the merge base of #23636, so applying that PR on top shows exactly what it changes. ## What changes are included in this PR? A new `datafusion/sqllogictest/test_files/functional_dependencies.slt` with one section per consumer of functional dependencies: 1. `ReplaceDistinctWithAggregate` — removing `DISTINCT` 2. `eliminate_duplicated_expr` — dropping trailing `ORDER BY` keys 3. `optimize_projections` — dropping `GROUP BY` expressions 4. `add_group_by_exprs_from_dependencies` — selecting non-grouped columns 5. `GROUP BY` derived keys on the NULL-padded side of an outer join Each section contrasts a non-nullable `PRIMARY KEY` against a nullable `UNIQUE` column, because that is the distinction the consumers get wrong: SQL `UNIQUE` permits multiple `NULL` rows, so a `UNIQUE` column is a key only among the non-NULL rows. Section 5 uses no constraints at all, so it isolates the outer-join interaction from the `UNIQUE` cases. Cases that currently return wrong answers are labelled `BUG` with the expected result and a link to the issue: | Case | Symptom | Issue | | --- | --- | --- | | 1.2 | `DISTINCT` over a nullable `UNIQUE` column returns both `NULL` rows | #23634 | | 2.2 | `ORDER BY x, y` drops the `y` key, so the `NULL` rows come back unordered | #23818 | | 3.2 | `GROUP BY x, y` drops `y`, merging the two `NULL` groups and losing a row | #23819 | | 4.2 | `SELECT x, y ... GROUP BY x` returns two rows for the `x = NULL` group | #23820 | Only logical plans are shown, since all of these rules run during logical optimization. ## Are these changes tested? This PR is only tests. `cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- functional_dependencies.slt` passes. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
