[
https://issues.apache.org/jira/browse/FLINK-41054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Diljeet Singh updated FLINK-41054:
----------------------------------
Description:
InternalSourceSplitMetricGroup registers seven per-split metrics under
split.<splitId>.watermark.*: currentWatermark, activeTimeMsPerSecond,
pausedTimeMsPerSecond, idleTimeMsPerSecond, accumulatedActiveTimeMs,
accumulatedPausedTimeMs and accumulatedIdleTimeMs. When a split finishes,
SourceOperator#splitFinished closes the group. That unregisters the metrics on
the TaskManager, so they no longer appear in its MetricQueryService dump.
The JobManager's MetricStore only merges each fetched dump into its existing
maps. It removes whole jobs, TaskManagers, subtasks and old attempts, but never
an individual metric that stops being reported. The finished splits' metrics
therefore stay in the store with their last values. As a result, endpoints such
as /jobs/:jobid/vertices/:vertexid/subtasks/metrics (and the subtask and
attempt metric endpoints) list a growing number of metrics for each split ever
processed. For sources with many short-lived splits, such as FileSource in
continuous monitoring mode, this grows without bound. It increases JobManager
memory use, makes REST responses slower, and clutters the metric picker in the
web UI with stale entries.
Steps to reproduce:
1. Run a job with a FLIP-27 source that processes many splits over time, such
as a FileSource in continuous monitoring mode.
2. Call GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics repeatedly.
3. The returned list keeps growing with split.<id>.watermark.* entries for
splits that have already finished.
was:
InternalSourceSplitMetricGroup registers seven per-split metrics under
split.<splitId>.watermark.*: currentWatermark, activeTimeMsPerSecond,
pausedTimeMsPerSecond, idleTimeMsPerSecond, accumulatedActiveTimeMs,
accumulatedPausedTimeMs and accumulatedIdleTimeMs. When a split finishes,
SourceOperator#splitFinished closes the group. That unregisters the metrics on
the TaskManager, so they no longer appear in its MetricQueryService dump.
The JobManager's MetricStore only merges each fetched dump into its existing
maps. It removes whole jobs, TaskManagers, subtasks and old attempts, but never
an individual metric that stops being reported. The finished splits' metrics
therefore stay in the store with their last values. As a result, endpoints such
as /jobs/:jobid/vertices/:vertexid/subtasks/metrics (and the subtask and
attempt metric endpoints) list a growing number of metrics for each split ever
processed. For sources with many short-lived splits, such as FileSource in
continuous monitoring mode, streaming reads from Iceberg or Paimon, or CDC
snapshot chunks, this grows without bound. It increases JobManager memory use,
makes REST responses slower, and clutters the metric picker in the web UI with
stale entries.
Steps to reproduce:
1. Run a job with a FLIP-27 source that processes many splits over time, such
as a FileSource in continuous monitoring mode.
2. Call GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics repeatedly.
3. The returned list keeps growing with split.<id>.watermark.* entries for
splits that have already finished.
> Metrics of finished source splits are never removed from the REST API's
> metric store
> ------------------------------------------------------------------------------------
>
> Key: FLINK-41054
> URL: https://issues.apache.org/jira/browse/FLINK-41054
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Metrics, Runtime / REST
> Affects Versions: 2.3.0, 2.2.1, 2.1.3, 2.3.1
> Reporter: Diljeet Singh
> Assignee: Diljeet Singh
> Priority: Major
> Labels: pull-request-available
>
> InternalSourceSplitMetricGroup registers seven per-split metrics under
> split.<splitId>.watermark.*: currentWatermark, activeTimeMsPerSecond,
> pausedTimeMsPerSecond, idleTimeMsPerSecond, accumulatedActiveTimeMs,
> accumulatedPausedTimeMs and accumulatedIdleTimeMs. When a split finishes,
> SourceOperator#splitFinished closes the group. That unregisters the metrics
> on the TaskManager, so they no longer appear in its MetricQueryService dump.
> The JobManager's MetricStore only merges each fetched dump into its existing
> maps. It removes whole jobs, TaskManagers, subtasks and old attempts, but
> never an individual metric that stops being reported. The finished splits'
> metrics therefore stay in the store with their last values. As a result,
> endpoints such as /jobs/:jobid/vertices/:vertexid/subtasks/metrics (and the
> subtask and attempt metric endpoints) list a growing number of metrics for
> each split ever processed. For sources with many short-lived splits, such as
> FileSource in continuous monitoring mode, this grows without bound. It
> increases JobManager memory use, makes REST responses slower, and clutters
> the metric picker in the web UI with stale entries.
> Steps to reproduce:
> 1. Run a job with a FLIP-27 source that processes many splits over time, such
> as a FileSource in continuous monitoring mode.
> 2. Call GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics repeatedly.
> 3. The returned list keeps growing with split.<id>.watermark.* entries for
> splits that have already finished.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
