[
https://issues.apache.org/jira/browse/BEAM-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16478847#comment-16478847
]
Etienne Chauchot commented on BEAM-3926:
----------------------------------------
Hi Alex,
Thanks for your comments. You're right the PR includes the reference java impl
for MetrisPusher that, indeed, aggregates himself the metrics at each thread
tick with a call to runner-core asAttemptedMetrics() and then pushes the result
to an external sink. But for dataflow, there will be a different C++ impl of
MetricsPusher (we discussed it yesterday evening with Pablo) that can be
completely different and that will reside in the actual dataflow engine. I'm
not sure I understand what you mean by "service", MetricsPusher aims to be
instanciated as close as possible from the workers (for spark it is
instanciated in the driver) and to push to external sinks like a REST backend
or a Ganglia server. These backends do not aggregate worker metrics value but
expect to receive the current value of the metric no matter what the workers
topology is. That being said, if dataflow needs to have a service inside its
ecosystem to receive un-aggregated metrics and then aggregate them, fair
enough, it is an implementation detail. The important is that at user
configured period, aggregated metrics are sent to the configured external sink.
And that external sink can be hosted outside of the dataflow environment.
IMHO reusing the existing java MetricsPusher for dataflow can be more complex
than coding a C++ one. All it needs to do is:
* read options provided by the user (what sink to use, what frequency to push,
additional sink related parameters) available in the serialized pipelineOptions
received with the job
* schedule a thread that push metrics to the configured metrics sink (or use
your aggregation service).
* at each thread tick, check the state of the pipeline and if it is in
terminal state, then terminate the pushing thread.
* The simplest for pipeline isolation is to instanciate a MetricsPusher per
pipeline.
> Support MetricsPusher in Dataflow Runner
> ----------------------------------------
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
> Reporter: Scott Wegner
> Assignee: Pablo Estrada
> Priority: Major
>
> See [relevant email
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
> From [~echauchot]:
>
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very
> different from the others. It just submits a job to the cloud hosted engine.
> So, no access to metrics container etc... from the runner. So I think that
> the MetricsPusher (component responsible for merging metrics and pushing them
> to a sink backend) must not be instanciated in DataflowRunner otherwise it
> would be more a client (driver) piece of code and we will lose all the
> interest of being close to the execution engine (among other things
> instrumentation of the execution of the pipelines). I think that the
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
