danielnadean commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5236790893
## Design proposal (for discussion on this issue)
Following the same pattern adopted on #11666 (see discussion there with
@nzw921rx and @DanielLeens): keeping pre-implementation design alignment here
on the issue instead of a standalone docs PR. User-facing docs will be updated
together with the implementation PR once the feature ships, not merged ahead of
it. Closing the earlier docs-only PR (#11732) accordingly.
### Existing foundation (why this can stay small)
| Area | Existing contract |
|---|---|
| Per-node JVM/host metrics | `GET /system-monitoring-information` already
returns ~35 fields per node (CPU load, heap/physical memory, GC, thread count,
...). The Workers/Master page renders only 4 of them today. |
| Per-worker slot state | `ResourceManager#getRegisterWorker()` returns a
live `ConcurrentMap<Address, WorkerProfile>`. Each `WorkerProfile` already
carries `assignedSlots`/`unassignedSlots` (`SlotProfile[]`) and
`systemLoadInfo` (`cpuPercentage`/`memPercentage`) - the same state the
scheduler itself uses to accept/reject allocation requests. |
| Cross-node read pattern | `GetOverviewOperation`/`OverviewService` already
show the established local-vs-forward-to-master routing for reaching this state
from REST. |
So the "per-worker slot accounting" this issue originally asked for isn't
new bookkeeping - it already exists inside `WorkerProfile`. The only new
backend work is a read-only projection of it.
### V1 scope
- Extend the Workers/Master table with a curated column set (Host, Port,
Role, CPU Load, Heap Used/Max, Physical MEM, GC, Threads, Slots used/total)
instead of today's 4 columns.
- Re-enable the existing (currently commented-out) Action column as a "View
Details" drawer showing the full raw monitoring payload, so no field is lost by
curating the table.
- New read-only `GET /resource/workers` endpoint projecting `WorkerProfile`
-> `{host, port, totalSlot, usedSlot, dynamicSlot, cpuPercentage,
memPercentage, attributes}`. No new field on `WorkerProfile`, no change to any
allocation/scheduling code path.
- Client-side join between `/system-monitoring-information` and
`/resource/workers` by `host:port`.
### Non-goals for V1
- Per-worker "tasks currently running here" via `/trace/task-mapping` - that
endpoint is scoped per-job; a cluster-wide, cross-job view needs its own
fan-out design and is left for a follow-up.
- Historical retention of node resource metrics, capacity
planning/autoscaling.
### Validation
- Unit test coverage for the `WorkerProfile` -> projection mapping,
including a worker with no slots/load reported yet (must not throw).
- Frontend test coverage for the client-side join, including an unmatched
row.
Implementation PR: #11733 (draft; docs bundled in the same PR per the above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
