, Edward
Sent: Monday, December 21, 2020 12:32 PM
To: Yun Gao ; user@flink.apache.org
Subject: RE: RE: checkpointing seems to be throttled.
Doh! Yeah, we set the state backend in code and I read the flink-conf.yaml
file and use the high-availability storage dir.
From: Yun Gao mailto:yungao
Doh! Yeah, we set the state backend in code and I read the flink-conf.yaml
file and use the high-availability storage dir.
From: Yun Gao
Sent: Monday, December 21, 2020 11:28 AM
To: Colletta, Edward ; user@flink.apache.org
Subject: Re: RE: checkpointing seems to be throttled.
This email
--
Sender:Colletta, Edward
Send Date:Tue Dec 22 00:04:03 2020
Recipients:Yun Gao , user@flink.apache.org
Subject:RE: checkpointing seems to be throttled.
Thanks for the quick response.
We are using FsStateBackend, and I did see checkpoint files and directories in
the EFS mounted directory
; user@flink.apache.org
Subject: Re: checkpointing seems to be throttled.
This email is from an external source - exercise caution regarding links and
attachments.
Hi Edward,
For the second issue, have you also set the statebackend type? I'm asking
so because except for the default heap
Hi Edward,
For the second issue, have you also set the statebackend type? I'm asking
so because except for the default heap statebackend, other statebackends should
throws exception if the state.checkpoint.dir is not set. Since heap
statebackend stores all the snapshots in the JM's memory,
Using session cluster with three taskmanagers, cluster.evenly-spread-out-slots
is set to true. 13 jobs running. Average parallelism of each job is 4.
Flink version 1.11.2, Java 11.
Running on AWS EC2 instances with EFS for high-availability.storageDir.
We are seeing very high checkpoint times
