Martijn Visser created FLINK-40648:
--------------------------------------
Summary: PrometheusReporter drops all metrics of a process when
two metric group variables sanitise to the same label name
Key: FLINK-40648
URL: https://issues.apache.org/jira/browse/FLINK-40648
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics
Affects Versions: 1.20.5, 2.2.1, 2.3.0, 2.4.0
Reporter: Martijn Visser
AbstractPrometheusReporter.notifyOfAddedMetric builds the Prometheus label
names by filtering every metric group variable through [^a-zA-Z0-9:_] -> _ and
adding each result to dimensionKeys, without checking whether two keys produced
the same name.
metricGroup.addGroup("a.b", "v1").addGroup("a-b", "v2") gives the variables
<a.b> and <a-b>, both of which filter to a_b, and the scrape then carries
{code}
flink_scope_dup{a_b="v1",a_b="v2",} 7.0
{code}
Prometheus 3.11.2 scraping that body:
{code}
health = down
lastError= label name "a_b" is not unique: invalid sample
{code}
The rejection is in tsdb/head_append.go, and scrape.go has no case for
ErrInvalidSample, so it breaks out of the sample loop. The scrape is abandoned
rather than the one sample skipped, which means every other metric from that
TaskManager or JobManager is lost too and the target goes down. Verified with a
body holding the colliding metric plus one valid metric: both come back empty
and up is 0.
Only a user-supplied addGroup key can trigger this. Flink's own variables are
all [a-z0-9_] already.
Two metrics named a.b and a-b in one group collide the same way one level up.
There the second metric reuses the first one's collector, so the series shows
whichever registered last under the first one's help string. That is the same
defect FLINK-32649 reports, since collectorsWithCountByMetricName is keyed on
the scoped name without the label names.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
