DanielLeens opened a new issue, #11665: URL: https://github.com/apache/seatunnel/issues/11665
## Background Zeta's REST API already returns rich per-node monitoring data via `GET /system-monitoring-information`: roughly 35 fields including CPU load (process/system/systemAverage), heap/physical/managed memory, GC counts and time, and thread counts (the full payload is already modeled in `seatunnel-engine-ui/src/service/manager/types.ts`). However, the Workers/Master page (`seatunnel-engine-ui/src/views/managers/index.tsx`) only renders 4 of those fields today - Host, Port, Physical MEM Total, and Heap MEM Used - and an Action column is present in the source but commented out. Separately, Zeta already computes a full task-to-worker assignment (`GET /trace/task-mapping/:jobId`, backed by `TaskMappingBuilder`) with per-task host/worker binding, but that endpoint has no UI consumer at all today. What is genuinely missing, not just unrendered, is per-worker **slot accounting**: `OverviewInfo` only exposes cluster-wide `totalSlot`/`unassignedSlot` integers; there is no per-node slot total/used breakdown anywhere in the current model. In Flink's Web UI, the "Task Managers" tab is the equivalent first-class page: per-TM memory, CPU, GC, and slot occupancy, plus the tasks currently scheduled on it. ## Problem to solve Job-level and pipeline-level observability (#11351/#11352/#11353) can tell you *that* a vertex is slow, but not *why* at the infrastructure layer - e.g. whether the host running that task is CPU-saturated, low on heap, or has no free slots left for retry/redeploy. Today that requires shelling into a node directly, even though most of the raw data is already being collected by the cluster itself. ## Proposed scope Extend the existing Workers/Master page into a proper node resource view, mostly by surfacing data that already exists: - render the full already-fetched `/system-monitoring-information` payload (CPU load, GC, thread count, etc.) instead of the current 4 columns - add per-worker slot accounting (total/used) to the node model and surface it in the same view - this is the one part that needs new server-side bookkeeping, since slots are currently only tracked cluster-wide - wire the existing `/trace/task-mapping` endpoint into this page so a worker row can expand into "tasks currently running here", linking back to the owning job - keep this primarily a UI + light REST-extension change, since the bulk of the raw resource data is already collected ## Why this should be a dedicated feature This is cluster/infrastructure-level observability, distinct from job/pipeline-level observability: - it is scoped by node, not by job or vertex, so it needs its own UI treatment and a small REST extension (per-worker slots) - it must keep working (and stay useful) even when no job is currently running - per-worker slot accounting must stay consistent with the live scheduler view, which is a correctness-sensitive contract, not just a display change ## STIP requirement before implementation Because this adds new per-worker slot-accounting state, **the claimant should submit a STIP design first and get maintainer agreement before starting implementation**. The STIP can stay narrow given how much already exists: - how per-worker slot total/used is derived and kept consistent with the scheduler's live view - whether `/trace/task-mapping` needs any changes to serve a per-worker "tasks running here" list efficiently, or is already sufficient as-is - refresh interval and cost on larger clusters for the now-fuller `/system-monitoring-information` payload ## Acceptance criteria - The Workers/Master page renders the full monitoring payload already returned by `/system-monitoring-information`, not just today's 4 fields. - Users can see per-worker slot usage and the tasks currently assigned to each worker, without SSHing into a host. - The view remains useful when the cluster is idle (no jobs running). - REST/API and UI behavior are documented clearly. - English and Chinese docs are updated. ## Non-goals for the first version - historical/long-term retention of node resource metrics - cluster capacity planning or autoscaling recommendations - per-node log tailing (already covered by the existing worker log viewer) ## Related work - Issue #11351 - Issue #11352 - Issue #11353 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
