yusufyildiz34 opened a new pull request, #29665:
URL: https://github.com/apache/flink/pull/29665
## What is the purpose of the change
In batch mode, a DISTINCT or GROUP BY over an outer join can drop a row when
a grouping column of the null-padded input is NULL. For the query from the
JIRA, `SELECT DISTINCT * FROM (... FULL JOIN ...)` returns only `(NULL, NULL,
NULL)` instead of `(NULL, 1.0, NULL), (NULL, NULL, NULL)`.
`FlinkRelMdUniqueGroups.getUniqueGroups(Join)` falls back to the union of
the unique groups of both inputs, regardless of the join type. On the
null-padded input of an outer join, a padded row falls into the same group as a
real row whose unique group columns are NULL. The unique groups then no longer
determine the other columns of that input, and `AggregateReduceGroupingRule`
turns those columns into invalid auxiliary groups.
## Brief change log
- `FlinkRelMdUniqueGroups` only keeps the unique groups of a null-padded
input if they contain a column that is never NULL on the real rows. That is
either a NOT NULL column, or, on the inner side of a LEFT/RIGHT join, a key of
a null-rejecting equi-condition. Otherwise it keeps all requested columns of
that input.
- As a result, batch aggregates over an outer join no longer use auxiliary
groups for a null-padded input whose unique key may be NULL on matched rows.
Among the existing plan tests, only `testAggOnFullJoin1/2` change.
`AggregateReduceGroupingRule` is batch-only, so streaming plans don't change.
The affected queries now group on more columns, giving up the optimization for
correct results.
A related issue in the same method, the handling of non-equi join
conditions, is tracked in FLINK-41200 and will be fixed in a separate PR on top
of this one.
## Verifying this change
This change added tests and can be verified as follows:
- `AggregateReduceGroupingITCase#testDistinctOnFullJoinWithNullGroupingKey`
reproduces the JIRA scenario with a NULL grouping key. Without the fix it
returns 7 instead of 8 rows.
- `AggregateReduceGroupingITCase#testDistinctOnLeftJoinWithNullGroupingKey`
covers the same issue for a left join, where a matched and a padded row share a
NULL grouping key. Without the fix it returns 3 instead of 4 rows.
-
`FlinkRelMdUniqueGroupsTest#testGetUniqueGroupsOnOuterJoinWithNullPaddedInput`
covers:
- a left join on a non-key column and one on `IS NOT DISTINCT FROM`, where
the groups are not reduced
- a full join on a NOT NULL unique key, where the groups are still reduced
- `FlinkRelMdUniqueGroupsTest#testGetUniqueGroupsOnJoin` changes its
expectations for full joins, and for a right join without equi-condition,
because their unique keys are nullable. The old expectations assumed that a
nullable unique key of a null-padded input still determines its other columns.
- The golden plans of `testAggOnFullJoin1/2` in
`AggregateReduceGroupingTest` and `AggregateReduceGroupingRuleTest` are
regenerated without auxiliary groups. No other plan changed. All planner plan
tests (`org.apache.flink.table.planner.plan.**`, excluding restore and semantic
tests) and the batch aggregate / join ITCases pass.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
