Thanks Robert for your answer, this seems to be what we observed when we tried to delete the first time: Flink complained about missing files. I'm wondering then how are people cleaning their storage for incremental checkpoints? Is there any guarantee when using TTLs that after the TTL has expired, no more file older than the TTL will be needed in the shared folder?
Le mar. 3 août 2021 à 13:29, Robert Metzger <rmetz...@apache.org> a écrit : > Hi Robin, > > Let's say you have two checkpoints #1 and #2, where #1 has been created by > an old version or your job, and #2 has been created by the new version. > When can you delete #1? > In #1, there's a directory "/shared" that contains data that is also used > by #2, because of the incremental nature of the checkpoints. > > You can not delete the data in the /shared directory, as this data is > potentially still in use. > > I know this is only a partial answer to your question. I'll try to find > out more details and extend my answer later. > > > On Thu, Jul 29, 2021 at 2:31 PM Robin Cassan < > robin.cas...@contentsquare.com> wrote: > >> Hi all! >> >> We've happily been running a Flink job in production for a year now, with >> the RocksDB state backend and incremental retained checkpointing on S3. We >> often release new versions of our jobs, which means we cancel the running >> one and submit another while restoring the previous jobId's last retained >> checkpoint. >> >> This works fine, but we also need to clean old files from S3 which are >> starting to pile up. We are wondering two things: >> - once the newer job has restored the older job's checkpoint, is it safe >> to delete it? Or will the newer job's checkpoints reference files from the >> older job, in which case deleting the old checkpoints might cause errors >> during the next restore? >> - also, since all our state has a 7 days TTL, is it safe to set a 7 or 8 >> days retention policy on S3 which would automatically clean old files, or >> could we still need to retain files older than 7 days even with the TTL? >> >> Don't hesitate to ask me if anything is not clear enough! >> >> Thanks, >> Robin >> >
