Hi all, We hit FLINK-37686 in production last week on Flink 2.2.1 (ForSt disaggregated state on S3, primary-dir at the default checkpoint dir, num-retained 3) and I want to raise it here because the impact was severe and the code path is unchanged on master.
The pattern was deterministic: after every restore the job completed exactly num-retained checkpoints, then every checkpoint failed in enableFileDeletions with FileNotFoundException until the failure threshold forced a failover, which started the next cycle. We went through six of these cycles in one afternoon on a ~6.4 TB job, losing 30 to 50 minutes of reprocessing per cycle. The loop only broke once ingest dropped in the evening, to me this seems to indicate that the failure is a race between the JM deleting shared handlesand the TM-side file mapping purging its entries. This would explain why it “self healed” during lower traffic, the number of at-risk files scales with write volume. I added a comment to the ticket with the full analysis, logs, and a timeline. The TLDR; the first failure happens exactly when the restored checkpoint leaves the retained set, the listStatus full scan HEADs every mapping entry and dies on the first deleted object, and nothing else can drop the stale entry, so the failure repeats until failover. The relevant classes are identical across release-2.2.1, release-2.3.0 and master, so I believe every 2.x release is affected. I would like to suggest a fix: tolerate FileNotFoundException for NOT_OWNED entries in the purge-scan listStatus path and drop them from the mapping. Does that direction sound right, or is a broader rework of file ownership planned here? Ticket: https://issues.apache.org/jira/browse/FLINK-37686 Thanks, Francis
