Leeeee16 opened a new issue, #6573: URL: https://github.com/apache/seatunnel/issues/6573
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened During testing, as I don't have access to an HDFS server, I use the local file system for storage. When i use MySQL CDC, storing checkpoints on the local file system results in errors. While synchronizing smaller tables (fewer than 1000 records), the process executes without any problems. When synchronizing larger datasets (a table with around 200,000 records), I experience errors. Despite increasing the checkpoint timeout setting, but I still get this error. ### SeaTunnel Version 2.3.3 ### SeaTunnel Config ```conf seatunnel: engine: history-job-expire-minutes: 1440 backup-count: 1 queue-type: blockingqueue print-execution-info-interval: 60 print-job-metrics-info-interval: 60 slot-service: dynamic-slot: true checkpoint: interval: 10000 timeout: 60000 storage: type: hdfs max-retained: 3 plugin-config: namespace: /tmp/seatunnel/checkpoint_snapshot storage.type: hdfs fs.defaultFS: file:///tmp/ # Ensure that the directory has written permission env { # You can set flink configuration here execution.parallelism = 2 job.mode = "STREAMING" checkpoint.interval = 2000 #execution.checkpoint.interval = 10000 #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint" } source { # This is a example source plugin **only for test and demonstrate the feature source plugin** MySQL-CDC { base-url = "jdbc:mysql://xxx:port/test?useUnicode=true&characterEncoding=utf-8&useSSL=false" parallelism = 2 username = "uuu" password = "xxx" table-names = ["test.aaa"] startup.mode = "initial" } # If you would like to get more information about how to configure SeaTunnel and see full list of source plugins, # please go to https://seatunnel.apache.org/docs/category/source-v2 } sink { jdbc { url = "jdbc:mysql://xxx:port/test2?useUnicode=true&characterEncoding=utf-8&useSSL=false" driver = "com.mysql.cj.jdbc.Driver" user = "uuu" password = "xxx" generate_sink_sql = true # You need to configure both database and table database = test2 table = aaa primary_keys = ["ID"] } #Console { #} # If you would like to get more information about how to configure SeaTunnel and see full list of sink plugins, # please go to https://seatunnel.apache.org/docs/category/sink-v2 } ``` ### Running Command ```shell ./bin/seatunnel.sh --config ./config/test.config.template ``` ### Error Exception ```log ... 2024-03-26 10:06:45,827 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 68 2024-03-26 10:06:47,822 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - timeout checkpoint: 824825597350903809/1/39, CHECKPOINT_TYPE 2024-03-26 10:06:47,823 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml 2024-03-26 10:06:47,823 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - trigger checkpoint failed org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:172) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$19(CheckpointCoordinator.java:645) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:643) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:261) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:532) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] 2024-03-26 10:06:47,823 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - trigger checkpoint failed org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:172) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$19(CheckpointCoordinator.java:645) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:643) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:261) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:532) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] 2024-03-26 10:06:47,823 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - trigger checkpoint failed org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:172) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$19(CheckpointCoordinator.java:645) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:643) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:261) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:532) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] 2024-03-26 10:06:47,824 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - trigger checkpoint failed org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:172) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$19(CheckpointCoordinator.java:645) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:643) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:261) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:532) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] 2024-03-26 10:06:47,825 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - trigger checkpoint failed org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:172) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$19(CheckpointCoordinator.java:645) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:643) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:261) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:532) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] 2024-03-26 10:06:47,825 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - trigger checkpoint failed org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.abortCheckpoint(PendingCheckpoint.java:172) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$cleanPendingCheckpoint$19(CheckpointCoordinator.java:645) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4707) ~[?:1.8.0_202] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.cleanPendingCheckpoint(CheckpointCoordinator.java:643) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:261) ~[seatunnel-starter.jar:2.3.3] at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:532) ~[seatunnel-starter.jar:2.3.3] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] ... 2024-03-26 10:06:47,851 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,853 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,853 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from RUNNING to FAILED 2024-03-26 10:06:47,853 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,853 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,853 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - start cancel job Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] count = 0 2024-03-26 10:06:47,853 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,854 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,857 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from FAILED to FAILED 2024-03-26 10:06:47,857 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,857 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,857 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] turn from state RUNNING to CANCELING. 2024-03-26 10:06:47,857 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,857 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from FAILED to FAILED 2024-03-26 10:06:47,857 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,858 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,858 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,858 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from FAILED to FAILED 2024-03-26 10:06:47,858 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,858 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,858 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,858 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from FAILED to FAILED 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] state from CREATED to CANCELED 2024-03-26 10:06:47,859 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] state in Imap is RUNNING, not equals expected state CREATED 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] state from SCHEDULED to CANCELED 2024-03-26 10:06:47,859 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] state in Imap is RUNNING, not equals expected state SCHEDULED 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] state from DEPLOYING to CANCELED 2024-03-26 10:06:47,859 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] state in Imap is RUNNING, not equals expected state DEPLOYING 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] state from RUNNING to CANCELING 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,859 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,859 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,860 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] state from CREATED to CANCELED 2024-03-26 10:06:47,860 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] state in Imap is RUNNING, not equals expected state CREATED 2024-03-26 10:06:47,860 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] state from SCHEDULED to CANCELED 2024-03-26 10:06:47,860 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] state in Imap is RUNNING, not equals expected state SCHEDULED 2024-03-26 10:06:47,860 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] state from DEPLOYING to CANCELED 2024-03-26 10:06:47,860 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] state in Imap is RUNNING, not equals expected state DEPLOYING 2024-03-26 10:06:47,860 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] state from RUNNING to CANCELING 2024-03-26 10:06:47,860 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from FAILED to FAILED 2024-03-26 10:06:47,862 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] state from CREATED to CANCELED 2024-03-26 10:06:47,862 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,862 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] state in Imap is RUNNING, not equals expected state CREATED 2024-03-26 10:06:47,862 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] state from SCHEDULED to CANCELED 2024-03-26 10:06:47,862 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] state in Imap is RUNNING, not equals expected state SCHEDULED 2024-03-26 10:06:47,862 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] state from DEPLOYING to CANCELED 2024-03-26 10:06:47,862 WARN org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] state in Imap is RUNNING, not equals expected state DEPLOYING 2024-03-26 10:06:47,862 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] state from RUNNING to CANCELING 2024-03-26 10:06:47,862 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,863 ERROR org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] cancel error with exception: java.lang.InterruptedException at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:347) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.seatunnel.engine.server.dag.physical.SubPlan.cancelPipelineTasks(SubPlan.java:461) at org.apache.seatunnel.engine.server.dag.physical.SubPlan.cancelPipeline(SubPlan.java:417) at org.apache.seatunnel.engine.server.dag.physical.SubPlan.handleCheckpointError(SubPlan.java:659) at org.apache.seatunnel.engine.server.master.JobMaster.lambda$handleCheckpointError$2(JobMaster.java:341) at java.util.ArrayList.forEach(ArrayList.java:1257) at org.apache.seatunnel.engine.server.master.JobMaster.handleCheckpointError(JobMaster.java:338) at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.handleCheckpointError(CheckpointManager.java:180) at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266) at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:532) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2024-03-26 10:06:47,863 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,863 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] turn from state RUNNING to CANCELING. 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from FAILED to FAILED 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] turn from state RUNNING to CANCELING. 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,864 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] turn from state RUNNING to CANCELING. 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_824825597350903809_1 state from FAILED to FAILED 2024-03-26 10:06:47,864 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,864 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Send cancel Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SplitEnumerator (1/1)] operator to member [SZJD-CSY-VM-0S01-awareness-es-001]:5801 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [SZJD-CSY-VM-0S01-awareness-es-001]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=824825597350903809, pipelineId=1, taskGroupId=1}) need cancel. 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Send cancel Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (1/2)] operator to member [SZJD-CSY-VM-0S01-awareness-es-001]:5801 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Send cancel Job SeaTunnel_Job (824825597350903809), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-MySQL-CDC-default-identifier]-SourceTask (2/2)] operator to member [SZJD-CSY-VM-0S01-awareness-es-001]:5801 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [SZJD-CSY-VM-0S01-awareness-es-001]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=824825597350903809, pipelineId=1, taskGroupId=30000}) need cancel. 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [SZJD-CSY-VM-0S01-awareness-es-001]:5801 [seatunnel] [5.1] Task (TaskGroupLocation{jobId=824825597350903809, pipelineId=1, taskGroupId=30001}) need cancel. 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error. 2024-03-26 10:06:47,865 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [SZJD-CSY-VM-0S01-awareness-es-001]:5801 [seatunnel] [5.1] Interrupted task 50001 - org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@22ae7e0 2024-03-26 10:06:47,865 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [SZJD-CSY-VM-0S01-awareness-es-001]:5801 [seatunnel] [5.1] taskDone, taskId = 50001, taskGroup = TaskGroupLocation{jobId=824825597350903809, pipelineId=1, taskGroupId=30001} 2024-03-26 10:06:47,865 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [SZJD-CSY-VM-0S01-awareness-es-001]:5801 [seatunnel] [5.1] Interrupted task 20000 - org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask@54f006b8 2024-03-26 10:06:47,865 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [SZJD-CSY-VM-0S01-awareness-es-001]:5801 [seatunnel] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@68f4e56f ``` ### Zeta or Flink or Spark Version Zeta 2.3.3 ### Java or Scala Version java1.8 ### Screenshots _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
