[
https://issues.apache.org/jira/browse/BEAM-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16343068#comment-16343068
]
Etienne Chauchot commented on BEAM-3310:
----------------------------------------
[~lcwik] I'm targetting point 3.b
{quote}
This could work as an extension to a pipeline if metrics sinks were defined
part of pipeline construction and then metrics events would be sent to SDKs
either on the control stream or over the data plane.
Alternatively this could be baked inside an SDK harness but would be limited
to only reporting metrics for that given SDK instead of across all SDKs.
{quote}
Metrics sinks are configured through pipeline options and instantiated when
{{{{MetricsPusher}}}} is instantiated by the runners. I did not want the user
to have anything to do in his pipeline code. The code of metricsPusher and
sinks are in runner-core. For now, there is nothing in the code related to
portability. I'll submit the PR within the week and ask you as a reviewer.
> Push metrics to a backend in an runner agnostic way
> ---------------------------------------------------
>
> Key: BEAM-3310
> URL: https://issues.apache.org/jira/browse/BEAM-3310
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: Major
>
> The idea is to avoid relying on the runners to provide access to the metrics
> (either at the end of the pipeline or while it runs) because they don't have
> all the same capabilities towards metrics (e.g. spark runner configures sinks
> like csv, graphite or in memory sinks using the spark engine conf). The
> target is to push the metrics in the common runner code so that no matter the
> chosen runner, a user can get his metrics out of beam.
> Here is the link to the discussion thread on the dev ML:
> https://lists.apache.org/thread.html/01a80d62f2df6b84bfa41f05e15fda900178f882877c294fed8be91e@%3Cdev.beam.apache.org%3E
> And the design doc:
> https://s.apache.org/runner_independent_metrics_extraction
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
