Alexander Bij created NIFI-16296:
------------------------------------
Summary: Stale Prometheus cluster_connected_node_count metrics
accumulate indefinitely due to unbounded connected_nodes label values
Key: NIFI-16296
URL: https://issues.apache.org/jira/browse/NIFI-16296
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 2.11.0, 2.10.0
Reporter: Alexander Bij
The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus)
exposes cluster_connected_node_count with a connected_nodes label containing a
human-readable ratio string, e.g. connected_nodes="3 / 4". Because this label's
value changes over the lifetime of the process (as nodes join/leave the
cluster) and the underlying Gauge is never cleared, every distinct ratio the
cluster has ever reported remains permanently exposed as a separate, frozen
time series -- alongside the current, correct one.
For a 4-node cluster this means a scrape can return multiple lines like:
{noformat}
cluster_connected_node_count\{instance="<uuid>",connected_nodes="3 / 4"} 3.0
cluster_connected_node_count\{instance="<uuid>",connected_nodes="4 / 4"} 4.0
{noformat}
...even though the cluster is currently fully connected (4/4). The "3 / 4"
series is stale -- a snapshot from an earlier, transient state -- but is never
removed, so it is scraped forever (until the node's JVM restarts), making it
impossible to build a reliable Prometheus/Grafana panel for cluster health
without manually filtering out stale label combinations you can't distinguish
from current ones.
h2. Root cause
ClusterMetricsRegistry (org.apache.nifi.prometheusutil.ClusterMetricsRegistry)
is a permanent singleton field on StandardNiFiServiceFacade:
{code:java}
private final ClusterMetricsRegistry clusterMetricsRegistry = new
ClusterMetricsRegistry();
{code}
Its cluster_connected_node_count gauge is declared with connected_nodes as a
*label*, not just a numeric value:
{code:java}
nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
.name("cluster_connected_node_count")
.help("The number of connected nodes in this cluster")
.labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
.register(registry));
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
