yux created FLINK-37368:
---------------------------
Summary: MySQL Pipeline job failed to checkpoint after closing
idle readers
Key: FLINK-37368
URL: https://issues.apache.org/jira/browse/FLINK-37368
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.3.0
Reporter: yux
Currently, PreTransformOperator uses `getUnionListState` to store upstream
schema maps. However, with MySQL `scan.incremental.close-idle-reader.enabled`
option, most subTasks will be closed (goes into "finished" state) after
snapshot stage finishes, which would block all later checkpoints:
{code:java}
/**
* If a job vertex using {@code UnionListState} has part of tasks FINISHED
where others are
* still in RUNNING state, the checkpoint would be aborted since it might cause
incomplete
* {@code UnionListState}.
*/
private void checkNoPartlyFinishedVertexUsedUnionListState(
Map<JobVertexID, ExecutionJobVertex> partlyFinishedVertex,
Map<OperatorID, OperatorState> operatorStates) {
for (ExecutionJobVertex vertex : partlyFinishedVertex.values()) {
if (hasUsedUnionListState(vertex, operatorStates)) {
throw new PartialFinishingNotSupportedByStateException(
String.format(
"The vertex %s (id = %s) has used"
+ " UnionListState, but part of its tasks
are FINISHED.",
vertex.getName(), vertex.getJobVertexId()));
}
}
} {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
