[ 
https://issues.apache.org/jira/browse/FLINK-37368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Leonard Xu reassigned FLINK-37368:
----------------------------------

    Assignee: yux

> MySQL Pipeline job failed to checkpoint after closing idle readers
> ------------------------------------------------------------------
>
>                 Key: FLINK-37368
>                 URL: https://issues.apache.org/jira/browse/FLINK-37368
>             Project: Flink
>          Issue Type: Bug
>          Components: Flink CDC
>    Affects Versions: cdc-3.3.0
>            Reporter: yux
>            Assignee: yux
>            Priority: Major
>
> Currently, PreTransformOperator uses `getUnionListState` to store upstream 
> schema maps. However, with MySQL `scan.incremental.close-idle-reader.enabled` 
> option, most subTasks will be closed (goes into "finished" state) after 
> snapshot stage finishes, which would block all later checkpoints:
> {code:java}
> /**
>  * If a job vertex using {@code UnionListState} has part of tasks FINISHED 
> where others are
>  * still in RUNNING state, the checkpoint would be aborted since it might 
> cause incomplete
>  * {@code UnionListState}.
>  */
> private void checkNoPartlyFinishedVertexUsedUnionListState(
>         Map<JobVertexID, ExecutionJobVertex> partlyFinishedVertex,
>         Map<OperatorID, OperatorState> operatorStates) {
>     for (ExecutionJobVertex vertex : partlyFinishedVertex.values()) {
>         if (hasUsedUnionListState(vertex, operatorStates)) {
>             throw new PartialFinishingNotSupportedByStateException(
>                     String.format(
>                             "The vertex %s (id = %s) has used"
>                                     + " UnionListState, but part of its tasks 
> are FINISHED.",
>                             vertex.getName(), vertex.getJobVertexId()));
>         }
>     }
> } {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to