[
https://issues.apache.org/jira/browse/NIFI-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Bij updated NIFI-16296:
---------------------------------
Description:
The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus)
exposes cluster_connected_node_count with a connected_nodes label containing a
human-readable ratio string, e.g. connected_nodes="3 / 4". Because this label's
value changes over the lifetime of the process (as nodes join/leave the
cluster) and the underlying Gauge is never cleared, every distinct ratio the
cluster has ever reported remains permanently exposed as a separate, frozen
time series – alongside the current, correct one.
For a 4-node cluster this means a scrape can return multiple lines like:
{noformat}
cluster_connected_node_count{instance="<uuid>",connected_nodes="3 / 4",} 3.0
cluster_connected_node_count{instance="<uuid>",connected_nodes="4 / 4",} 4.0
{noformat}
...even though the cluster is currently fully connected (4/4). The "3 / 4"
series is stale – a snapshot from an earlier, transient state – but is never
removed, so it is scraped forever (until the node's JVM restarts), making it
impossible to build a reliable Prometheus/Grafana panel for cluster health
without manually filtering out stale label combinations you can't distinguish
from current ones.
h2. Root cause
ClusterMetricsRegistry (org.apache.nifi.prometheusutil.ClusterMetricsRegistry)
is a permanent singleton field on StandardNiFiServiceFacade:
{code:java}
private final ClusterMetricsRegistry clusterMetricsRegistry = new
ClusterMetricsRegistry();
{code}
Its cluster_connected_node_count gauge is declared with connected_nodes as a
{*}label{*}, not just a numeric value:
{code:java}
nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
.name("cluster_connected_node_count")
.help("The number of connected nodes in this cluster")
.labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
.register(registry));
{code}
was:
The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus)
exposes cluster_connected_node_count with a connected_nodes label containing a
human-readable ratio string, e.g. connected_nodes="3 / 4". Because this label's
value changes over the lifetime of the process (as nodes join/leave the
cluster) and the underlying Gauge is never cleared, every distinct ratio the
cluster has ever reported remains permanently exposed as a separate, frozen
time series -- alongside the current, correct one.
For a 4-node cluster this means a scrape can return multiple lines like:
{noformat}
cluster_connected_node_count\{instance="<uuid>",connected_nodes="3 / 4"} 3.0
cluster_connected_node_count\{instance="<uuid>",connected_nodes="4 / 4"} 4.0
{noformat}
...even though the cluster is currently fully connected (4/4). The "3 / 4"
series is stale -- a snapshot from an earlier, transient state -- but is never
removed, so it is scraped forever (until the node's JVM restarts), making it
impossible to build a reliable Prometheus/Grafana panel for cluster health
without manually filtering out stale label combinations you can't distinguish
from current ones.
h2. Root cause
ClusterMetricsRegistry (org.apache.nifi.prometheusutil.ClusterMetricsRegistry)
is a permanent singleton field on StandardNiFiServiceFacade:
{code:java}
private final ClusterMetricsRegistry clusterMetricsRegistry = new
ClusterMetricsRegistry();
{code}
Its cluster_connected_node_count gauge is declared with connected_nodes as a
*label*, not just a numeric value:
{code:java}
nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
.name("cluster_connected_node_count")
.help("The number of connected nodes in this cluster")
.labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
.register(registry));
{code}
> Stale Prometheus cluster_connected_node_count metrics accumulate indefinitely
> due to unbounded connected_nodes label values
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16296
> URL: https://issues.apache.org/jira/browse/NIFI-16296
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.10.0, 2.11.0
> Reporter: Alexander Bij
> Priority: Major
> Labels: Matrices, monitoring, prometheus
>
> The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus)
> exposes cluster_connected_node_count with a connected_nodes label containing
> a human-readable ratio string, e.g. connected_nodes="3 / 4". Because this
> label's value changes over the lifetime of the process (as nodes join/leave
> the cluster) and the underlying Gauge is never cleared, every distinct ratio
> the cluster has ever reported remains permanently exposed as a separate,
> frozen time series – alongside the current, correct one.
>
> For a 4-node cluster this means a scrape can return multiple lines like:
> {noformat}
> cluster_connected_node_count{instance="<uuid>",connected_nodes="3 / 4",} 3.0
> cluster_connected_node_count{instance="<uuid>",connected_nodes="4 / 4",} 4.0
> {noformat}
> ...even though the cluster is currently fully connected (4/4). The "3 / 4"
> series is stale – a snapshot from an earlier, transient state – but is never
> removed, so it is scraped forever (until the node's JVM restarts), making it
> impossible to build a reliable Prometheus/Grafana panel for cluster health
> without manually filtering out stale label combinations you can't distinguish
> from current ones.
>
> h2. Root cause
> ClusterMetricsRegistry
> (org.apache.nifi.prometheusutil.ClusterMetricsRegistry) is a permanent
> singleton field on StandardNiFiServiceFacade:
> {code:java}
> private final ClusterMetricsRegistry clusterMetricsRegistry = new
> ClusterMetricsRegistry();
> {code}
> Its cluster_connected_node_count gauge is declared with connected_nodes as a
> {*}label{*}, not just a numeric value:
> {code:java}
> nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
> .name("cluster_connected_node_count")
> .help("The number of connected nodes in this cluster")
> .labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
> .register(registry));
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
