yew1eb opened a new pull request, #3782: URL: https://github.com/apache/celeborn/pull/3782
### What changes were proposed in this pull request? Add two gauges on the worker to expose disk health status, registered in `LocalDeviceMonitor.init()` and evaluated lazily from the live `DiskInfo.status` at scrape time: 1. `UnhealthyDiskCount` — the number of local disks whose status is not `HEALTHY`, intended for alerting (e.g. `> 0`). 2. `DiskStatus` with a `mountpoint` label — the current `DiskStatus` value (0: HEALTHY, 1: READ_OR_WRITE_FAILURE, 2: IO_HANG, 3: HIGH_DISK_USAGE, 4: CRITICAL_ERROR) of each disk, for locating the exact unhealthy disk. Also included: two corresponding panels in the Grafana dashboard (`assets/grafana/celeborn-dashboard.json`) and the new metrics in `docs/monitoring.md`. ### Why are the changes needed? `DeviceMonitor` already maintains `DiskInfo.status` for every local disk, but none of the existing metrics expose it — `DeviceOSFreeBytes`/`DeviceCelebornFreeBytes` report capacity, and `Device_<name>_<Status>_Count` only counts non-critical error events. To notice a broken disk, operators have to grep worker logs, which makes alerting on disk failures impossible. ### Does this PR resolve a correctness bug? - [ ] Yes ### Does this PR introduce _any_ user-facing change? - [x] Yes Two new worker metrics are exposed: `UnhealthyDiskCount` and `DiskStatus` (with a `mountpoint` label). ### How was this patch tested? - Added a new test case `monitor disk status metrics` in `DeviceMonitorSuite` covering the multi-disk scenario: all disks healthy initially, then one disk marked `READ_OR_WRITE_FAILURE` (aggregate count becomes 1 and only that disk's gauge changes), then recovered to `HEALTHY`. - `./build/mvn test -pl worker` passes. - `python3 dev/lint_grafana.py assets/grafana/celeborn-dashboard.json` passes. - `./build/mvn spotless:check -pl worker` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
