SEZ9 opened a new issue, #12685: URL: https://github.com/apache/seatunnel/issues/12685
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description `seatunnel --diagnose <jobId>` (added recently) answers "why did my job fail" and "why is my job stuck", but it cannot answer the most common question asked about a long-running CDC job: > CDC sync was stable at first, then it started lagging. How do I find out why? For such a job the current rules are all silent by construction. The job is `RUNNING`, both row counters are non-zero, and `restoreCount` is 0, so the only output is: ``` Job 852 mysql-cdc-to-doris — RUNNING [info] Running for 72h00m. (state=RUNNING) ``` That is accurate and useless. The reason is structural: every existing rule compares against zero (`SourceReceivedCount == 0`) or against a fixed threshold (`restoreCount >= 2`, state held `> 120s`). Gradual throughput degradation is neither — the counters keep moving, just more slowly. What is missing is a comparison of the job against *its own recent past*. ### Proposal Add checkpoint-based rules to `--diagnose`, reading the existing `/jobs/checkpoints/{jobId}` endpoint. No engine change is needed; this endpoint already publishes a usable time series and is available by default. Per pipeline it returns `counts` (`triggered`/`completed`/`failed`/`inProgress`/`restored`), `latestCompleted`/`latestFailed`, an `inProgress[]` array and a `history[]` of up to 32 entries, each carrying `durationMillis`, `stateSize`, `triggerTimestamp` and `failureReason`. Rules this supports: 1. **Checkpoint duration growing** — compare the recent half of completed durations against the older half. A checkpoint that took 2s a day ago and takes 90s now is the direct mechanism behind "it got slower", and it is visible in a single request. 2. **Checkpoint stalled** — an `inProgress` entry whose `triggerTimestamp` is old and whose `acknowledged < total`. This names *which* subtasks have not acknowledged the barrier, so it points at a specific stage instead of the job as a whole. 3. **Checkpoints failing** — `counts.failed > 0`, reported with `latestFailed.failureReason`. Repeated checkpoint failure on a CDC job means offsets are not advancing even though rows appear to flow. 4. **No checkpoint ever completed** — `triggered > 0 && completed == 0`. 5. **Checkpoint state growing** — `stateSize` trend, as the usual cause of (1). Two details worth recording, because both fail silently if assumed rather than checked: - `history` is **newest-first** (`PipelineCheckpointOverview.addHistory` uses `addFirst`, and `CheckpointMonitorService.getHistory` sorts by `triggerTimestamp` descending). A trend computed in array order without accounting for this reports the exact opposite conclusion. - **Failed and canceled checkpoints also appear in `history`, without `durationMillis` or `completedTimestamp`** (those fields are omitted when null). A duration trend must filter on `status` first, or the missing values poison the result. ### Scope Read-only and model-free, like the existing `--diagnose` rules: one additional GET, no submit, no stop, no savepoint, no API key required. A follow-up will add root-cause attribution from `/metrics/realtime` (`bpRatio`, `queueFillRatio`, `sinkBusyRatio`, `sourceIdleRatio`), which can distinguish "the sink cannot keep up" from "the source is the bottleneck". That is deliberately kept separate because `engine.observability.enabled` defaults to `false` and its retention window is only 3 minutes (10 max), so it needs user action to be useful and cannot answer "how much slower than three days ago". ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
