[
https://issues.apache.org/jira/browse/HDDS-17264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17264:
------------------------------
Attachment: HDDS-17264.001.patch
> Add Grafana dashboard and Prometheus alert rules for datanode volume health
> ---------------------------------------------------------------------------
>
> Key: HDDS-17264
> URL: https://issues.apache.org/jira/browse/HDDS-17264
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Dashboards
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-17264.001.patch
>
>
> The DataNode Overview dashboard (HDDS-15324) sums volume I/O metrics per
> host, which hides a single slow or failing disk. No alert rules ship with the
> compose monitoring setup.
> Proposal:
> * Add an "Ozone - Datanode Volume Health" dashboard with:
> ** failed volumes per datanode, by volume type
> ** disk I/O errors per volume ({{NumDataPathIOErrors}}, from HDDS-17261)
> ** per-volume read/write p99 and average latency
> ** each volume's p99 relative to the median of the volumes on the same
> datanode
> ** per-volume read/write operation rates
> * Add {{compose/ozone/prometheus-rules.yml}}, loaded by the monitoring
> compose setup, with alerts for:
> ** any failed volume
> ** more than 3 disk I/O errors on a volume within an hour
> ** a volume whose read or write p99 stays above 100 ms and 5x the datanode
> median for 15 minutes
> Validated with {{promtool check rules}} and {{promtool test rules}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
