Diljeet Singh created FLINK-41054:
-------------------------------------
Summary: Metrics of finished source splits are never removed from
the REST API's metric store
Key: FLINK-41054
URL: https://issues.apache.org/jira/browse/FLINK-41054
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics, Runtime / REST
Affects Versions: 2.1.3, 2.2.1, 2.3.0, 2.3.1
Reporter: Diljeet Singh
InternalSourceSplitMetricGroup registers per-split metrics under
split.<splitId>.watermark.*, for example currentWatermark, pausedTimePerSecond,
idleTimePerSecond and activeTimePerSecond. When a split finishes,
SourceOperator#splitFinished closes the group. That unregisters the metrics on
the TaskManager, so they no longer appear in its MetricQueryService dump.
The JobManager's MetricStore only merges each fetched dump into its existing
maps. It removes whole jobs, TaskManagers, subtasks and old attempts, but never
an individual metric that stops being reported. The finished splits' metrics
therefore stay in the store with their last values. As a result, endpoints such
as /jobs/:jobid/vertices/:vertexid/subtasks/metrics (and the subtask and
attempt metric endpoints) list a growing number of metrics for each split ever
processed. For sources with many short-lived splits, such as file or Kafka
sources with partition discovery, this grows without bound. It increases
JobManager memory use, makes REST responses slower, and clutters the metric
picker in the web UI with stale entries.
Steps to reproduce:
1. Run a job with a FLIP-27 source that processes many splits over time, such
as a FileSource in continuous monitoring mode.
2. Call GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics repeatedly.
3. The returned list keeps growing with split.<id>.watermark.* entries for
splits that have already finished.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
