[
https://issues.apache.org/jira/browse/FLINK-40997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146678#comment-18146678
]
Gustavo de Morais commented on FLINK-40997:
-------------------------------------------
Hey [~fcsaky] [~mateczagany], thanks for the quick backports and trying to fix
it across branches.
FLINK-41216 is only in master and 2.4, so on 2.3, 2.2 and 1.20 a join behind a
Top-N that outputs the row number now always picks the broken key. As I
mentioned, It's a narrow case, but it's wrong every time now. Do you want to
backport it to 2.3?
A comment on older branches backport: we usually go with the last two releases
and max the 1.20 LTS when it's a critical fix. See [update
policy|https://flink.apache.org/downloads/#update-policy-for-old-releases].
Since both fixes change plans for re-submitted SQL, I'd stop at 2.3 for
FLINK-41216. That's also we do other important fixes. The rule is a bit relaxed
and if you think a fix is very important there is always the case to be made
that it has to be backported to other versions as well. In this case, if you
opt to do it, it makes sense to bring the other related bug.
> Regular join picks a random state key among equal-length upsert keys
> --------------------------------------------------------------------
>
> Key: FLINK-40997
> URL: https://issues.apache.org/jira/browse/FLINK-40997
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Affects Versions: 2.0.2, 2.1.3, 2.2.2, 1.20.6, 2.3.1
> Reporter: Mate Czagany
> Assignee: Mate Czagany
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.4.0, 1.20.7, 2.2.3, 2.3.2
>
>
> The same SQL can produce different {{leftUpsertKeys}} / {{rightUpsertKeys}}
> order on {{StreamExecJoin}} nodes.
> If the input has two or more upsert keys that tie for the fewest columns, and
> none of them is contained in that side's join key, otherwise
> {{JoinKeyContainsUniqueKey}} would be used.
> Example:
> {code:sql}
> CREATE TABLE items (id INT, score INT, brand STRING, PRIMARY KEY (id) NOT
> ENFORCED) WITH (...);
> CREATE TABLE z (k STRING, z STRING) WITH (...);
> INSERT INTO snk
> SELECT t.id, t.rn, z.z
> FROM (SELECT * FROM (SELECT *, ROW_NUMBER() OVER (ORDER BY score DESC) AS rn
> FROM items)
> WHERE rn <= 3) t
> JOIN z ON t.brand = z.k;
> {code}
> The left input of the join is {{{}(id, brand, rn){}}}. It has two upsert
> keys, {{id}} and {{{}rn{}}}, and the join key {{brand}} contains neither.
> After compiling this several times, {{leftUpsertKeys}} of the join switches
> between [[0],[2]] (state keyed by {{{}id{}}}) and [[2],[0]] (keyed by
> {{{}rn{}}}). This can break state restore using a previous snapshot with a
> {{{}StateMigrationException{}}}.
> With id {{BIGINT}} instead of {{{}INT{}}}, both keys are {{{}BIGINT NOT
> NULL{}}}, a state restore can succeed. Retractions then look up the wrong
> key, and stale rows stay in the join state and keep joining. No error is
> reported.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)