[
https://issues.apache.org/jira/browse/BEAM-6172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712160#comment-16712160
]
Micah Wylde commented on BEAM-6172:
-----------------------------------
In addition to those two changes, there's an extra "group" in the name. For
example, a group-specific native flink metric (as output from the slf4j
reporter with default configs) looks like
{code}
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1206234804-16211aec.group.0.numRecordsIn:
41
{code}
while a beam one looks like
{code}
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1206234804-16211aec.group.0.__counter__group__org.apache.beam.runners.core.ReduceFnRunner__droppedDueToClosedWindow:
41
{code}
I think ideally the beam metric would be
{code}
10.100.208.242.taskmanager.4f13adf64e7315b7198911465ca44119.BeamApp-mwylde-1206234804-16211aec.group.0.org.apache.beam.runners.core.ReduceFnRunner.droppedDueToClosedWindow:
41
{code}
Our high-level goal is to be able to use the same metric parsing logic as we
use for existing flink metrics (both internal and application-specific) and get
reasonable results back.
> Flink metrics are not generated in standard format
> --------------------------------------------------
>
> Key: BEAM-6172
> URL: https://issues.apache.org/jira/browse/BEAM-6172
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Affects Versions: 2.8.0
> Reporter: Micah Wylde
> Assignee: Maximilian Michels
> Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The metrics that the flink runner exports do not follow the standard format
> used by Flink, and doesn't respect Flink metric configuration options.
> For example (with the default metrics configuration) beam produces a metric:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.group.0.__counter__group__org-apache-beam-runners-core-ReduceFnRunner__droppedDueToClosedWindow
> {code}
> whereas a native Flink metric looks like:
> {code}
> 10-100-209-71.taskmanager.0f29b420b63fea58f6f321bc0cbf45f3.BeamApp-mwylde-1203224439-a7d8fdf6.Source-Custom-Source-7Kinesis-None-beam-env-docker-v1-0-ToKeyedWorkItem.0.numRecordsOut
> {code}
> In particular, Beam should respect the
> [metric.scope.delimiter|https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/config.html#metrics-scope-delimiter]
> configuration for separating components of a metric (currently it uses
> "__"), and should not include the type of metric (counter, gauge, etc.).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
