[
https://issues.apache.org/jira/browse/HDDS-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642217#comment-17642217
]
Neil Joshi commented on HDDS-7576:
----------------------------------
Problem uncovered and discussed in
https://github.com/apache/ozone/pull/3781#issuecomment-1306594864
> Prometheus metrics do not remove stale metrics until restart
> ------------------------------------------------------------
>
> Key: HDDS-7576
> URL: https://issues.apache.org/jira/browse/HDDS-7576
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Neil Joshi
> Priority: Major
>
> Currently, the ozone prometheus metrics sink retains all metrics pushed by
> the MetricsSystem until it is reset. This results in Prometheus pulling and
> displaying from the sink metrics that are stale and unused. Should a metric
> be collected and pushed by the MetricsSystem that is active and in use for
> only a short period, that metric currently will by retained by the sink and
> pulled and displayed by Prometheus until the system is reset. This results
> in displaying stale metrics to the user.
> MetricSystem Prometheus sinks incorrectly retaining stale metrics is a
> problem that was reported and fixed in the Hadoop Common in
> https://issues.apache.org/jira/browse/HADOOP-17804. Similarly for the ozone
> PrometheusMetricsSink, the internal map used to store the metrics can be
> exposed upon a flush and reset (cleared) after the flush. This way only the
> active metrics are exposed to Prometheus from the sink. See description in
> https://github.com/apache/hadoop/pull/3369.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
