[
https://issues.apache.org/jira/browse/FLINK-40269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Fan updated FLINK-40269:
----------------------------
Fix Version/s: 2.0.3
1.20.6
> Unaligned checkpoint restore may fail after rescale when a two-input task
> reads from the same upstream
> ------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40269
> URL: https://issues.apache.org/jira/browse/FLINK-40269
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Reporter: Rui Fan
> Assignee: Rui Fan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.3, 2.2.2, 1.20.6, 2.3.1, 2.4.0, 2.1.4
>
>
> We found a restore failure in Flink 2.3.0 when restoring from an unaligned
> checkpoint after rescaling the job.
> This issue likely exists since the early unaligned-checkpoint-rescaling
> support around Flink 1.13/1.14.
> It can happen when one two-input task has both inputs derived from the same
> upstream stream:
>
> {code:java}
> source
> |-- rebalance() --> CoMap input 1
> |-- keyBy(...) --> CoMap input 2{code}
>
>
> During rescale restore, Flink can confuse the two input edges and route
> channel state with the wrong mapping. The job then fails during restore with
> an error like:
>
> {code:java}
> text Cannot select SubtaskConnectionDescriptor{inputSubtaskIndex=1,
> outputSubtaskIndex=0};
> known channels are [SubtaskConnectionDescriptor{inputSubtaskIndex=0,
> outputSubtaskIndex=0}]{code}
>
>
> Reproducer branch:
> https://github.com/1996fanrui/flink/tree/repro/unaligned-channel-remap
>
> Commit `2711b807112e50b4bdabbd1ca181f800e8968a15` adds a minimal ITCase that
> reproduces the failure 100% before the runtime fix:
> UnalignedCheckpointRescaleSameUpstreamITCase.java
>
> The proposed fix is in commit `9b8560e69818e25ac20e19875aca55bdea208fcc`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
