[ https://issues.apache.org/jira/browse/FLINK-37368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leonard Xu reassigned FLINK-37368: ---------------------------------- Assignee: yux > MySQL Pipeline job failed to checkpoint after closing idle readers > ------------------------------------------------------------------ > > Key: FLINK-37368 > URL: https://issues.apache.org/jira/browse/FLINK-37368 > Project: Flink > Issue Type: Bug > Components: Flink CDC > Affects Versions: cdc-3.3.0 > Reporter: yux > Assignee: yux > Priority: Major > > Currently, PreTransformOperator uses `getUnionListState` to store upstream > schema maps. However, with MySQL `scan.incremental.close-idle-reader.enabled` > option, most subTasks will be closed (goes into "finished" state) after > snapshot stage finishes, which would block all later checkpoints: > {code:java} > /** > * If a job vertex using {@code UnionListState} has part of tasks FINISHED > where others are > * still in RUNNING state, the checkpoint would be aborted since it might > cause incomplete > * {@code UnionListState}. > */ > private void checkNoPartlyFinishedVertexUsedUnionListState( > Map<JobVertexID, ExecutionJobVertex> partlyFinishedVertex, > Map<OperatorID, OperatorState> operatorStates) { > for (ExecutionJobVertex vertex : partlyFinishedVertex.values()) { > if (hasUsedUnionListState(vertex, operatorStates)) { > throw new PartialFinishingNotSupportedByStateException( > String.format( > "The vertex %s (id = %s) has used" > + " UnionListState, but part of its tasks > are FINISHED.", > vertex.getName(), vertex.getJobVertexId())); > } > } > } {code} > > -- This message was sent by Atlassian Jira (v8.20.10#820010)