JNSimba opened a new pull request, #63194:
URL: https://github.com/apache/doris/pull/63194
### What problem does this PR solve?
Related PR: #62224 (per-job metrics), #62269 (Lag column)
Problem Summary:
#62224 introduced per-job metrics (`streaming_job_per_job_scanned_rows`,
`_load_bytes`, `_filtered_rows`, `_succeed_task_count`, `_failed_task_count`)
for streaming insert jobs, exposed via `/metrics` with `job_id`/`job_name`
labels for Prometheus.
#62269 later added a `Lag` column to `SHOW JOBS` / `jobs()` TVF that reports
end-to-end CDC delay in seconds, but the value was only exposed through SQL —
there was no corresponding Prometheus metric, so dashboards/alerting on lag was
not possible.
This PR adds `streaming_job_per_job_lag` (unit: `SECONDS`) to the existing
per-job metric set, sourced from `SourceOffsetProvider.getLag()` via a new
`StreamingInsertJob#getLag()` wrapper. Empty lag (S3 / snapshot phase, not
applicable) is intentionally not registered so Prometheus does not carry a
misleading series for those jobs; idle CDC sources report `0`.
### Release note
Add `streaming_job_per_job_lag` Prometheus metric (seconds) for streaming
insert jobs with CDC sources.
### Check List (For Author)
- Test
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
