Hi Vinicius, Your case, the taskmanager being actively killed by yarn was the other way this happened.
You are using RocksDBStateBackend, right? Not being sure, I’ve got the strong suspicion that this has got to do with the glibc bug that is seemingly in the works. There is some documentation here [1] and a solution that has been implemented for k8s containers [2] which replaces the glibc allocator with libjemalloc.so . However we are not completely through with our encounter of the same problem. Our intermediate solution is to reserve some unused extra memory, so the problem is delayed but not completely prevented (we restart our jobs daily by means of savepoint taking): flink-conf.yaml: … taskmanager.memory.managed.fraction: 0.2 #reserve 2GB extra unused space (out of 8GB per TM) in order to mitigate the glibc memory leakage problem taskmanager.memory.task.off-heap.size: 2048mb … I’m not sure if and starting with which Flink version libjemalloc.so is integrated by default into the flink runtime … Flink team to the rescue 😊! Hope this helps Thias [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/mem_trouble/#container-memory-exceeded [2] https://issues.apache.org/jira/browse/FLINK-19125 From: Vinicius Peracini <vinicius.perac...@zenvia.com> Sent: Mittwoch, 9. März 2022 17:56 To: Schwalbe Matthias <matthias.schwa...@viseca.ch> Cc: Dawid Wysakowicz <dwysakow...@apache.org>; user@flink.apache.org Subject: Re: Could not stop job with a savepoint So apparently the YARN container for Task Manager is running out of memory during the savepoint execution. Never had any problems with checkpoints though. Task Manager configuration: "taskmanager.memory.process.size": "10240m", "taskmanager.memory.managed.fraction": "0.6", "taskmanager.memory.jvm-overhead.fraction": "0.07", "taskmanager.memory.jvm-metaspace.size": "192mb", "taskmanager.network.memory.buffer-debloat.enabled": "true", On Wed, Mar 9, 2022 at 1:33 PM Vinicius Peracini <vinicius.perac...@zenvia.com<mailto:vinicius.perac...@zenvia.com>> wrote: Bom dia Schwalbe! Thanks for the reply. I'm using Flink 1.14.0. EMR is a managed cluster platform to run big data applications on AWS. This way Flink services are running on YARN. I tried to create another savepoint today and was able to retrieve the Job Manager log: 2022-03-09 15:42:10,294 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Triggering savepoint for job 6f9d71e57efba96dad7f5328ab9ac717. 2022-03-09 15:42:10,298 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 1378 (type=SAVEPOINT) @ 1646840530294 for job 6f9d71e57efba96dad7f5328ab9ac717. 2022-03-09 15:45:19,636 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [/172.30.0.169:57520<http://172.30.0.169:57520>] failed with java.io.IOException: Connection reset by peer 2022-03-09 15:45:19,648 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639] has failed, address is now gated for [50] ms. Reason: [Disassociated] 2022-03-09 15:45:19,652 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink-metrics@ip-172-30-0-169.ec2.internal:41533] has failed, address is now gated for [50] ms. Reason: [Disassociated] 2022-03-09 15:45:19,707 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_BULK -> Map (1/3) (866e32468227f9f0adac82e9b83b970a) switched from RUNNING to FAILED on container_1646341714746_0005_01_000004 @ ip-172-30-0-165.ec2.internal (dataPort=40231). org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connection unexpectedly closed by remote task manager 'ip-172-30-0-169.ec2.internal/172.30.0.169:34413<http://172.30.0.169:34413>'. This might indicate that the remote task manager was lost. at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:186) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelInactive(NettyMessageClientDecoderDelegate.java:94) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[flink-dist_2.12-1.14.0.jar:1.14.0] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322] 2022-03-09 15:45:19,720 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 5d739cfcb34ba91e39db0d6db0a4f1a2_0. 2022-03-09 15:45:19,721 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 18 tasks should be restarted to recover the failed task 5d739cfcb34ba91e39db0d6db0a4f1a2_0. 2022-03-09 15:45:19,723 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job TOOL_MESSAGE_STREAM (6f9d71e57efba96dad7f5328ab9ac717) switched from state RUNNING to RESTARTING. 2022-03-09 15:45:19,728 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_BULK -> Map (3/3) (df8df89abf1761a726dd4593387cbd76) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,732 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 1 of source Source: BULK_SENDER_EVENT -> Filter -> Map. 2022-03-09 15:45:19,732 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: BULK_SENDER_EVENT -> Filter -> Map (2/3) (2ba29adbeeceb86a69536082fbfb4931) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,732 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_BULK -> Map (2/3) (3674e936354727457b06293592b65f20) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 of source Source: BULK_SENDER_EVENT -> Filter -> Map. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: BULK_SENDER_EVENT -> Filter -> Map (1/3) (117be79102d086ddb65fb0b31c245f49) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_FLOW -> Map (3/3) (3add641adeebe4e14dd0111bd647aa75) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_FLOW -> Map (2/3) (f5f4d0d49c1d2c2c65685ccb6c35eab4) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (3/3) (07a1c10037596788aee6844603ab17a2) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_FLOW -> Map (1/3) (3caaf3a8072b29567d94691d1765a294) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (2/3) (216e7b0a7d0f8c887a5c91a3e5267c73) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (1/3) (2e3973007d7c3d71f10456c3aca5a0b3) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)) (3/3) (bcebf365a3e6da0281d80b3ab0e2cff8) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,733 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 2 of source Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)). 2022-03-09 15:45:19,734 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 1 of source Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)). 2022-03-09 15:45:19,734 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)) (2/3) (8d912317f46bf11f866bd70be09377ef) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,734 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 of source Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)). 2022-03-09 15:45:19,734 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)) (1/3) (c4a8b6f2d0b7ea837dd4e69a107e652e) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 2 of source Source: FLOW_EVENT -> Filter -> Map. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: FLOW_EVENT -> Filter -> Map (3/3) (b811833b1ce35416fae61aba7cdbeb53) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 1 of source Source: FLOW_EVENT -> Filter -> Map. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: FLOW_EVENT -> Filter -> Map (2/3) (c5048ba45046caa3b954257602b4f0a4) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 0 of source Source: FLOW_EVENT -> Filter -> Map. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: FLOW_EVENT -> Filter -> Map (1/3) (4693404a84c87a07b1f35ed62f9edad7) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Removing registered reader after failure for subtask 2 of source Source: BULK_SENDER_EVENT -> Filter -> Map. 2022-03-09 15:45:19,735 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: BULK_SENDER_EVENT -> Filter -> Map (3/3) (d4421c51bf8fb716c672e13fd249450f) switched from RUNNING to CANCELING. 2022-03-09 15:45:19,744 WARN akka.remote.transport.netty.NettyTransport [] - Remote connection to [null] failed with java.net.ConnectException: Connection refused: ip-172-30-0-169.ec2.internal/172.30.0.169:46639<http://172.30.0.169:46639> 2022-03-09 15:45:19,746 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639]] Caused by: [java.net.ConnectException: Connection refused: ip-172-30-0-169.ec2.internal/172.30.0.169:46639<http://172.30.0.169:46639>] 2022-03-09 15:45:19,751 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_BULK -> Map (3/3) (df8df89abf1761a726dd4593387cbd76) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,751 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution df8df89abf1761a726dd4593387cbd76. 2022-03-09 15:45:19,753 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution df8df89abf1761a726dd4593387cbd76. 2022-03-09 15:45:19,754 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - LEFT_JOIN_MESSAGE_FLOW -> Map (3/3) (3add641adeebe4e14dd0111bd647aa75) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 3add641adeebe4e14dd0111bd647aa75. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 3add641adeebe4e14dd0111bd647aa75. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (3/3) (07a1c10037596788aee6844603ab17a2) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution 07a1c10037596788aee6844603ab17a2. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)) (3/3) (bcebf365a3e6da0281d80b3ab0e2cff8) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution bcebf365a3e6da0281d80b3ab0e2cff8. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution bcebf365a3e6da0281d80b3ab0e2cff8. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: FLOW_EVENT -> Filter -> Map (3/3) (b811833b1ce35416fae61aba7cdbeb53) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution b811833b1ce35416fae61aba7cdbeb53. 2022-03-09 15:45:19,755 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution b811833b1ce35416fae61aba7cdbeb53. 2022-03-09 15:45:19,756 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: BULK_SENDER_EVENT -> Filter -> Map (3/3) (d4421c51bf8fb716c672e13fd249450f) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution d4421c51bf8fb716c672e13fd249450f. 2022-03-09 15:45:19,757 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Discarding the results produced by task execution d4421c51bf8fb716c672e13fd249450f. 2022-03-09 15:45:19,757 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 6f9d71e57efba96dad7f5328ab9ac717: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=2}] 2022-03-09 15:45:19,790 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: BULK_SENDER_EVENT -> Filter -> Map (1/3) (117be79102d086ddb65fb0b31c245f49) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,797 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: FLOW_EVENT -> Filter -> Map (1/3) (4693404a84c87a07b1f35ed62f9edad7) switched from CANCELING to CANCELED. 2022-03-09 15:45:19,801 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, Filter)) (1/3) (c4a8b6f2d0b7ea837dd4e69a107e652e) switched from CANCELING to CANCELED. 2022-03-09 15:45:20,026 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (1/3) (2e3973007d7c3d71f10456c3aca5a0b3) switched from CANCELING to CANCELED. 2022-03-09 15:45:23,957 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker container_1646341714746_0005_01_000003 is terminated. Diagnostics: Container container_1646341714746_0005_01_000003 marked as failed. Exit code:137. Diagnostics:[2022-03-09 15:45:19.639]Container killed on request. Exit code is 137 [2022-03-09 15:45:19.641]Container exited with a non-zero exit code 137. [2022-03-09 15:45:19.642]Killed by external signal 2022-03-09 15:45:23,957 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Closing TaskExecutor connection container_1646341714746_0005_01_000003 because: Container container_1646341714746_0005_01_000003 marked as failed. Exit code:137. Diagnostics:[2022-03-09 15:45:19.639]Container killed on request. Exit code is 137 [2022-03-09 15:45:19.641]Container exited with a non-zero exit code 137. [2022-03-09 15:45:19.642]Killed by external signal Thanks, On Tue, Mar 8, 2022 at 4:57 AM Schwalbe Matthias <matthias.schwa...@viseca.ch<mailto:matthias.schwa...@viseca.ch>> wrote: Bom Dia Vinicius, Can You still find (and post) the exception stack from your jobmanager log, the flink client log does not reveal enough information. Your situation reminds me of something similar I had. In the log you might find something like this or similar: 2022-03-07 02:15:41,347 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Triggering stop-with-savepoint for job e12f22653f79194863ab426312dd666a. 2022-03-07 02:15:41,380 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 4983974 (type=SAVEPOINT_SUSPEND) @ 1646615741347 for job e12f22653f79194863ab426312dd666a. 2022-03-07 02:15:43,042 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Decline checkpoint 4983974 by task 0e659ac720e3e0b3e4072dbc1cc85cd3 of job e12f22653f79194863ab426312dd666a at container_e1093_1646358077201_0002_01_000001 @ ulxxphaddtn02.adgr.net<http://ulxxphaddtn02.adgr.net> (dataPort=44767). org.apache.flink.util.SerializedThrowable: Asynchronous task checkpoint failed. at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:279) ~[flink-dist_2.11-1.13.0.jar:1.13.0] BTW what Flink version are you running? What is EMR (what technology underneath). From: Vinicius Peracini <vinicius.perac...@zenvia.com<mailto:vinicius.perac...@zenvia.com>> Sent: Montag, 7. März 2022 20:46 To: Dawid Wysakowicz <dwysakow...@apache.org<mailto:dwysakow...@apache.org>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Could not stop job with a savepoint Hi Dawid, thanks for the reply. The job was still in progress and producing events. Unfortunately I was not able to stop the job with a savepoint or to just create a savepoint. I had to stop the job without the savepoint and restore the state using the last checkpoint. Still reviewing my configuration and trying to figure out why this is happening. Any help would be appreciated. Thanks! On Mon, Mar 7, 2022 at 11:56 AM Dawid Wysakowicz <dwysakow...@apache.org<mailto:dwysakow...@apache.org>> wrote: Hi, From the exception it seems the job has been already done when you're triggering the savepoint. Best, Dawid On 07/03/2022 14:56, Vinicius Peracini wrote: Hello everyone, I have a Flink job (version 1.14.0 running on EMR) and I'm having this issue while trying to stop a job with a savepoint on S3: org.apache.flink.util.FlinkException: Could not stop with a savepoint job "df3a3c590fabac737a17f1160c21094c". at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581) at org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002) at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CompletionException: org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator is suspending. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579) ... 9 more I'm using incremental and unaligned checkpoints (aligned checkpoint timeout is 30 seconds). I also tried to create the savepoint without stopping the job (using flink savepoint command) and got the same error. Any idea what is happening here? Thanks in advance, Aviso Legal: Este documento pode conter informações confidenciais e/ou privilegiadas. Se você não for o destinatário ou a pessoa autorizada a receber este documento, não deve usar, copiar ou divulgar as informações nele contidas ou tomar qualquer ação baseada nessas informações. Disclaimer: The information contained in this document may be privileged and confidential and protected from disclosure. If the reader of this document is not the intended recipient, or an employee agent responsible for delivering this document to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. Aviso Legal: Este documento pode conter informações confidenciais e/ou privilegiadas. Se você não for o destinatário ou a pessoa autorizada a receber este documento, não deve usar, copiar ou divulgar as informações nele contidas ou tomar qualquer ação baseada nessas informações. Disclaimer: The information contained in this document may be privileged and confidential and protected from disclosure. If the reader of this document is not the intended recipient, or an employee agent responsible for delivering this document to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited. Aviso Legal: Este documento pode conter informações confidenciais e/ou privilegiadas. Se você não for o destinatário ou a pessoa autorizada a receber este documento, não deve usar, copiar ou divulgar as informações nele contidas ou tomar qualquer ação baseada nessas informações. Disclaimer: The information contained in this document may be privileged and confidential and protected from disclosure. If the reader of this document is not the intended recipient, or an employee agent responsible for delivering this document to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
