Jozef Vilcek created BEAM-5246:
----------------------------------
Summary: Beam metrics exported as flink metrics are not correct
Key: BEAM-5246
URL: https://issues.apache.org/jira/browse/BEAM-5246
Project: Beam
Issue Type: Bug
Components: runner-flink
Affects Versions: 2.6.0
Reporter: Jozef Vilcek
Assignee: Aljoscha Krettek
In Flink UI and fink native MetricReported, I am seeing too many instances of
my Beam metric counter. It looks like the counter is materialised for every
operator running within the task, although is is emitter from only one beam
step (which should map to one operator?). This produces double counting.
A bit debugging I noticed this is happening for stream jobs. In batch I was not
able to reproduce it. Problem might be in FlinkMetricContainer.
[https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L86]
The update seems to be called from operators after finishing the bundle. Data
from accumulator are flushed to `runtimeContext.getMetricGroup()`. The scope of
accumulator seems to be different than metricGroup as in there with different
call the scope components change, especially for operatorID. It seems like
during the run, `metricResult.getStep()` does not match operatorName of
metricGroup where metric is being pushed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
