Neil Joshi created HDDS-7576:
--------------------------------
Summary: Prometheus metrics do not remove stale metrics until
restart
Key: HDDS-7576
URL: https://issues.apache.org/jira/browse/HDDS-7576
Project: Apache Ozone
Issue Type: Bug
Reporter: Neil Joshi
Currently, the ozone prometheus metrics sink retains all metrics pushed by the
MetricsSystem until it is reset. This results in Prometheus pulling and
displaying from the sink metrics that are stale and unused. Should a metric be
collected and pushed by the MetricsSystem that is active and in use for only a
short period, that metric currently will by retained by the sink and pulled and
displayed by Prometheus until the system is reset. This results in displaying
stale metrics to the user.
MetricSystem Prometheus sinks incorrectly retaining stale metrics is a problem
that was reported and fixed in the Hadoop Common in
https://issues.apache.org/jira/browse/HADOOP-17804. Similarly for the ozone
PrometheusMetricsSink, the internal map used to store the metrics can be
exposed upon a flush and reset (cleared) after the flush. This way only the
active metrics are exposed to Prometheus from the sink. See description in
https://github.com/apache/hadoop/pull/3369.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
