fanrui created FLINK-26049:
------------------------------
Summary: The tolerable-failed-checkpoints logic is invalid when
checkpoint trigger failed
Key: FLINK-26049
URL: https://issues.apache.org/jira/browse/FLINK-26049
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.14.3, 1.13.5
Reporter: fanrui
Fix For: 1.15.0
Attachments: image-2022-02-09-18-08-17-868.png,
image-2022-02-09-18-08-34-992.png, image-2022-02-09-18-08-42-920.png
After triggerCheckpoint, if checkpoint failed, flink will execute the
tolerable-failed-checkpoints logic. But if triggerCheckpoint failed, flink
won't execute the tolerable-failed-checkpoints logic.
h1. How to reproduce this issue?
In our online env, hdfs sre deletes the flink base dir by mistake, and flink
job don't have permission to create checkpoint dir. So cause flink trigger
checkpoint failed.
There are some didn't meet expectations:
* JM just log _"Failed to trigger checkpoint for job
6f09d4a15dad42b24d52c987f5471f18 since Trigger checkpoint failure" ._ Don't
show the root cause or exception.
* user set tolerable-failed-checkpoints=0, but if triggerCheckpoint failed,
flink won't execute the tolerable-failed-checkpoints logic.
* When triggerCheckpoint failed, numberOfFailedCheckpoints is always 0
* When triggerCheckpoint failed, we can't find checkpoint info in checkpoint
history page.
!image-2022-02-09-18-08-17-868.png!
!image-2022-02-09-18-08-34-992.png!
!image-2022-02-09-18-08-42-920.png!
h3. *All metrics are normal, so the next day we found out that the checkpoint
failed, and the checkpoint has been failing for a day. it's not acceptable to
the flink user.*
I have some ideas:
# Should tolerable-failed-checkpoints logic be executed when triggerCheckpoint
fails?
# When triggerCheckpoint failed, should increase numberOfFailedCheckpoints?
# When triggerCheckpoint failed, should show checkpoint info in checkpoint
history page?
# JM just show "Failed to trigger checkpoint", should we show detailed
exception to easy find the root cause?
Masters, could we do these changes? Please correct me if I'm wrong.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
