[
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=137733&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137733
]
ASF GitHub Bot logged work on BEAM-3310:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Aug/18 08:40
Start Date: 24/Aug/18 08:40
Worklog Time Spent: 10m
Work Description: echauchot commented on issue #4548: [BEAM-3310] Metrics
pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-415693529
@StephanEwen for the context please see the diagram above: basically Flink
accumulators are used to store `MetricsContainersStepMaps` that gather beam
metrics for the pipeline. When Flink merges these accumulators, Beam metrics
get merged. All the point of this discussion is the fact that these flink
accumulators are not available to the user in detached mode and thus Beam
methods that request Beam metrics do not work.
That being said, Beam Flink runner also updates internal Flink metrics via
the DoFnRunner: see
https://github.com/apache/beam/blob/0299676889dbc2db7ebb40db4acd27189735b75e/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/DoFnRunnerWithMetricsUpdate.java#L93
and
https://github.com/apache/beam/blob/0299676889dbc2db7ebb40db4acd27189735b75e/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L101
So I guess we can have access to Beam metrics from the native flink metrics
system in the Flink UI (not Beam `PipelineResult.metrics()` or `MetricsPusher`)
even in detached mode. Alternatively @aljoscha suggests to set up a Metrics
reporter
(https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html)
and Beam metrics will then be exported like Flink metrics to that reporter.
@JozoVilcek As I don't know native Flink well I cannot provide more details but
it may be redundant with the runner updating flink metrics (see links above).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 137733)
Time Spent: 16h 20m (was: 16h 10m)
> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
> Issue Type: New Feature
> Components: runner-extensions-metrics, sdk-java-core
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
> Time Spent: 16h 20m
> Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics
> (either at the end of the pipeline or while it runs) because they don't have
> all the same capabilities towards metrics (e.g. spark runner configures sinks
> like csv, graphite or in memory sinks using the spark engine conf). The
> target is to push the metrics in the common runner code so that no matter the
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML:
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
