Yang Li created FLINK-40238:
-------------------------------
Summary: Expose numberOfConsecutiveFailedCheckpoints as a
JobManager-level gauge
Key: FLINK-40238
URL: https://issues.apache.org/jira/browse/FLINK-40238
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing, Runtime / Metrics
Reporter: Yang Li
h2. Proposal
Add a new JobManager-level gauge alongside {{numberOfFailedCheckpoints}}:
{noformat}
numberOfConsecutiveFailedCheckpoints
{noformat}
The gauge mirrors {{CheckpointFailureManager}}'s internal counter:
increments on each checkpoint failure counted toward
{{tolerable-failed-checkpoints}}, resets on the next successful
checkpoint. Users see the same number the JobManager itself uses to
decide whether to fail the job, and the distance to
JobManager-initiated failover is simply
{{tolerable-failed-checkpoints - <gauge>}}.
Proposed scope:
{{DefaultCheckpointStatsTracker}}: register the new gauge and maintain
the counter via the {{CheckpointStatsListener}} event path (increment on
{{onFailedCheckpoint}} when the failure
reason is counted, reset on {{onCompletedCheckpoint}}).
No new config options, no public API or REST changes.
Doc update in {{docs/content/docs/ops/metrics.md}}, next to
{{numberOfFailedCheckpoints}}.
h2. Problem it solves
The core problem this solves is letting users detect that a job is in a bad
state
via external monitoring, by observing consecutive checkpoint failures.
This fires before the JobManager gives up, giving operators a window to
intervene. The alert predicate is textually the same as the config
value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
