[
https://issues.apache.org/jira/browse/BEAM-3310?focusedWorklogId=138207&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-138207
]
ASF GitHub Bot logged work on BEAM-3310:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Aug/18 15:30
Start Date: 26/Aug/18 15:30
Worklog Time Spent: 10m
Work Description: JozoVilcek commented on issue #4548: [BEAM-3310]
Metrics pusher
URL: https://github.com/apache/beam/pull/4548#issuecomment-416046976
Beam metrics are available through Flink UI and also being pushed to
`MetricReporter`. But Flink and Beam world of metrics are sort of disconnected
for the user. In Beam, metric is defined by `(metricType: [Counter,
Distribution, Gauge], namespace: String, name: String)` maybe also `stepName:
String` but that one is kind of internal to beam metrics system.
In flink `MetricReporter` world, user gets notification about added metrics
he can react to. They have form of `(metricType: Metric, metricName: String,
group: MetricGroup)` . This form us much richer and contains some extra
dimensions to beam model within `MetricGroup`. One have to be very much aware
to internals, to be able to work with this.
Metric pusher (and whole runner agnostic metrics idea) is currently the only
easy way how to conveniently retrieve aggregated metrics from Beam definition
domain. But, this feature is not always available, not in jobs launched in
detached mode. Therefore user have now only the option of implementing custom
Flink MetricReporter and extract metrics from there (requires a lot of internal
insight).
The question is, how can this be improved and have a convenience way of
reporting beam metrics also in detached jobs. I hope I do make sense and am not
confusing things too much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 138207)
Time Spent: 17h (was: 16h 50m)
> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
> Issue Type: New Feature
> Components: runner-extensions-metrics, sdk-java-core
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
> Time Spent: 17h
> Remaining Estimate: 0h
>
> The idea is to avoid relying on the runners to provide access to the metrics
> (either at the end of the pipeline or while it runs) because they don't have
> all the same capabilities towards metrics (e.g. spark runner configures sinks
> like csv, graphite or in memory sinks using the spark engine conf). The
> target is to push the metrics in the common runner code so that no matter the
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML:
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
