[
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=137945&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137945
]
ASF GitHub Bot logged work on BEAM-3310:
----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Aug/18 19:49
Start Date: 24/Aug/18 19:49
Worklog Time Spent: 10m
Work Description: zorro786 edited a comment on issue #4548: [BEAM-3310]
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-415863810
@aljoscha
Would like to respond to your email here to connect contexts:
> the accumulators not being available is an artifact of how detached mode
is implemented. Basically, we fire of the job and then forget everything about
it. That's why there is no way of getting any data about the job.
Regarding Metrics, with Flink you can set up a Metrics reporter
(https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html)
and Beam metrics will then be exported like Flink metrics to that reporter.
>Best,
Aljoscha
I understand that we forget about the job once we fire it in detached mode
(no way to get metrics from `PipelineResult`) and also that the Beam metrics
are available through a Metrics reporter from Flink.
Now my understanding was that these metrics from Beam need to be accumulated
by Flink first in order to report from across task managers etc. -> whether
through metrics reporter or Flink UI. This is what @StephanEwen assumed too. My
question is simple -> if Flink is actually accumulating metrics, why is it not
shown in Flink UI for Beam jobs under "Accumulators"? Is this a bug or this
support is not there from Flink? If support is not present then how are the
metrics reported/accumulated through Metrics reporter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 137945)
Time Spent: 16h 50m (was: 16h 40m)
> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
> Issue Type: New Feature
> Components: runner-extensions-metrics, sdk-java-core
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
> Time Spent: 16h 50m
> Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics
> (either at the end of the pipeline or while it runs) because they don't have
> all the same capabilities towards metrics (e.g. spark runner configures sinks
> like csv, graphite or in memory sinks using the spark engine conf). The
> target is to push the metrics in the common runner code so that no matter the
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML:
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
