manish1337 opened a new pull request, #29614: URL: https://github.com/apache/flink/pull/29614
## What is the purpose of the change A projection on top of `EXCEPT ALL` or `INTERSECT ALL` was pushed into both inputs and changed the result ([FLINK-41039](https://issues.apache.org/jira/browse/FLINK-41039)): ```sql SELECT y FROM ( SELECT x, y FROM (VALUES (1, 10)) AS a(x, y) EXCEPT ALL SELECT x, y FROM (VALUES (2, 10)) AS b(x, y)); -- expected: 10, actual: no rows (INTERSECT ALL: 10 instead of no rows) ``` Flink registers Calcite's `CoreRules.PROJECT_SET_OP_TRANSPOSE` to push a projection through a union, but the rule matches any `SetOp` and only skips distinct ones. `EXCEPT ALL` and `INTERSECT ALL` match rows on all of their fields, so pruning `x` makes the two rows match. Calcite 1.41's rule has the same problem (its own comment says union all "is the only operator this rule currently acts on"); I'll raise it upstream separately. ## Brief change log - Add `FlinkProjectSetOpTransposeRule`, which is Calcite's `ProjectSetOpTransposeRule` with the operand restricted to `LogicalUnion` - Register it instead of `CoreRules.PROJECT_SET_OP_TRANSPOSE` in `FlinkBatchRuleSets` and `FlinkStreamRuleSets` - Update the recorded plans of `batch.table.SetOperatorsTest#testProjectMinusTranspose`, which had recorded the wrong plan (grouped by `b, c` instead of `a, b, c`), and of the `EXCEPT ALL` / `INTERSECT ALL` cases of `PartialInsertTest` (same results, the projection now stays above the set operation) ## Verifying this change This change added tests and can be verified as follows: - Added `testMinusAllWithOuterProject` and `testIntersectAllWithOuterProject` to the batch and stream `SetOperatorsITCase`; they fail without the fix - Added `testMinusAllWithOuterProject`, `testIntersectAllWithOuterProject` and `testUnionAllWithOuterProject` to the batch and stream `SetOperatorsTest` plan tests; the first two fail without the fix, the union one checks the projection is still pushed into `UNION ALL` - Ran all planner plan tests (`org.apache.flink.table.planner.plan.**`) and the SQL and Table API `SetOperatorsITCase`s - Manually verified with a standalone cluster and `bin/sql-client.sh`, batch and streaming: without the fix the queries above return the wrong rows, with it they return the expected ones ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
