DanielLeens opened a new issue, #11668: URL: https://github.com/apache/seatunnel/issues/11668
## Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct). ## Search before asking - [x] I searched the existing issues and found no umbrella issue covering the full runtime-observability scope described below. The related work is currently tracked as separate, unlinked issues (#11351, #11352, #11353, #10750), plus three gaps filed alongside this umbrella (#11665, #11666, #11667). ## Describe the proposal SeaTunnel's Zeta engine is increasingly used as a drop-in replacement for Flink-based pipelines, and operators coming from Flink expect equivalent runtime observability: a live job graph with per-vertex/edge health, a backpressure panel, a checkpoint history/timeline, per-worker resource usage, live metric charts, a failure/exception history, and per-task data sampling - all reachable from the job's own page. Zeta's Web UI is already more capable here than a naive "Flink has it, we don't" comparison would suggest: the job detail page already renders a live DAG (built on AntV X6) with busy-ratio node coloring and backpressure/queue-fill edge coloring, polling `/metrics/realtime/jobs/{id}/vertices` and `.../edges` every 2 seconds. The recurring pattern across the gaps below is not "missing instrumentation" - it is **server-side data that already exists but is never surfaced in the UI, or internal state that is tracked but never read back**. That is a good sign for feasibility, and it is also exactly the kind of gap that is easy to lose track of across many separate issues. Seven issues currently exist that each cover one slice of this: - #11351 - a live runtime execution graph overlaying node/edge health on the job DAG. Already has an in-progress STIP/design PR (#11612) and a draft implementation PR (#11615). - #11352 - a dedicated backpressure diagnostics view ranking the worst bottlenecks. - #11353 - a checkpoint timeline with phase-level progress and failure breakdown. (Zeta already has a full checkpoint overview/history REST API - `/jobs/checkpoints/:jobId` and `/jobs/checkpoints/history/:jobId` - with zero UI consumption today.) - #10750 - viewing the latest N records per task vertex for running jobs, explicitly modeled on Flink/Spark's data-preview UX. - #11665 - a worker/node resource view. `/system-monitoring-information` already returns ~35 per-node metrics; the current Workers/Master page renders only 4 of them, and the existing `/trace/task-mapping` endpoint has no UI consumer at all. - #11666 - a selectable live metrics chart. The realtime endpoints already accept a `windowMs` parameter and return time-series data; today that data only appears as a raw table in a per-click drawer, never as a chart a user can pin and keep watching. - #11667 - a task exception/failure history. Today only the *first* error per sub-plan is kept (`PhysicalPlan.errorBySubPlan` uses `compareAndSet(null, ...)`) and there is no restart/attempt counter anywhere in the engine, even though per-task-group state-transition timestamps are already tracked internally and simply never read back. Each of these is independently useful and each already requires (or will require) its own STIP before implementation - this umbrella is not a request to redesign or replace any of their individual scopes or acceptance criteria. What is missing is the thing a maintainer already asked for on #11351: a single place that ties them together, so reviewers can evaluate the contracts that genuinely cut across all seven consistently (metric retention/sampling cost, refresh cadence, whether charting/graph components get shared) instead of re-deciding them independently per issue, and so new contributors can see at a glance what is already claimed (#11351) versus open for pickup. This also lines up with the project's own prior roadmap notes in `docs/en/engines/zeta/realtime-observability.md` (section 13, "Future Evolution Suggestions"), which already flags several of these directions (queueId-to-vertex DAG mapping, source fine-grained metrics, operator error/retry metrics) as known next steps. ## Task list - [ ] #11351 - runtime execution graph (STIP: #11612, implementation: #11615, both in progress) - [ ] #11352 - backpressure diagnostics view - [ ] #11353 - checkpoint timeline with phase-level breakdown - [ ] #10750 - latest-N-records preview per task vertex - [ ] #11665 - worker/node resource view - [ ] #11666 - selectable live metrics chart - [ ] #11667 - task exception/failure history ## Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
