[
https://issues.apache.org/jira/browse/FLINK-41039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129045#comment-18129045
]
Manish commented on FLINK-41039:
--------------------------------
Hi, I'd like to work on this. Could someone please assign it to me?
Proposed fix: Flink registers Calcite's CoreRules.PROJECT_SET_OP_TRANSPOSE in
both FlinkBatchRuleSets and FlinkStreamRuleSets with the comment "push project
through a Union", but the rule matches any SetOp. I plan to register a variant
whose operand is restricted to LogicalUnion, so a projection is no longer pushed
into the inputs of EXCEPT ALL / INTERSECT ALL. If the same problem exists in
Calcite itself, I'll also raise it upstream.
Tests: plan tests for EXCEPT ALL and INTERSECT ALL with a projection on top
(the projection must stay above the set operation), plus batch and streaming
ITCases with the queries from the description.
> A projection on top of EXCEPT ALL or INTERSECT ALL is pushed into both
> inputs and changes the result
> -----------------------------------------------------------------------------------------------------
>
> Key: FLINK-41039
> URL: https://issues.apache.org/jira/browse/FLINK-41039
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Affects Versions: 2.3.0
> Reporter: Yaoxuan Wu
> Priority: Major
>
> {code:sql}
> SELECT x, y FROM (VALUES (1, 10)) AS a(x, y)
> EXCEPT ALL
> SELECT x, y FROM (VALUES (2, 10)) AS b(x, y);
> -- (1, 10) correct
> SELECT y FROM (
> SELECT x, y FROM (VALUES (1, 10)) AS a(x, y)
> EXCEPT ALL
> SELECT x, y FROM (VALUES (2, 10)) AS b(x, y));
> -- expected: 10
> -- actual: (no rows)
> {code}
> {{ProjectSetOpTransposeRule}} pushes the projection {{y}} into both inputs,
> so the EXCEPT ALL is computed on {{y}} alone and the two rows cancel. The
> rule only refuses distinct set operations, but pushing a projection is only
> safe for UNION ALL. INTERSECT ALL is wrong in the same way (it returns rows
> that should not match). Streaming mode is affected too.
> Reproduces on 2.0.1, 2.2.1, 2.3.0 and master.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
