[
https://issues.apache.org/jira/browse/BEAM-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Wegner updated BEAM-3926:
-------------------------------
Description:
See [relevant email
thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
From [~echauchot]:
_AFAIK Dataflow being a cloud hosted engine, the related runner is very
different from the others. It just submits a job to the cloud hosted engine.
So, no access to metrics container etc... from the runner. So I think that the
MetricsPusher (component responsible for merging metrics and pushing them to a
sink backend) must not be instanciated in DataflowRunner otherwise it would be
more a client (driver) piece of code and we will lose all the interest of being
close to the execution engine (among other things instrumentation of the
execution of the pipelines). I think that the MetricsPusher needs to be
instanciated in the actual Dataflow engine._
was:
See relevant email thread [1]. From [~echauchot]:
{noformat}
AFAIK Dataflow being a cloud hosted engine, the related runner is very
different from the others. It just submits a job to the cloud hosted engine.
So, no access to metrics container etc... from the runner. So I think that the
MetricsPusher (component responsible for merging metrics and pushing them to a
sink backend) must not be instanciated in DataflowRunner otherwise it would be
more a client (driver) piece of code and we will lose all the interest of being
close to the execution engine (among other things instrumentation of the
execution of the pipelines). I think that the MetricsPusher needs to be
instanciated in the actual Dataflow engine. {noformat}
[1
]https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E
> Support MetricsPusher in Dataflow Runner
> ----------------------------------------
>
> Key: BEAM-3926
> URL: https://issues.apache.org/jira/browse/BEAM-3926
> Project: Beam
> Issue Type: Sub-task
> Components: runner-dataflow
> Reporter: Scott Wegner
> Assignee: Thomas Groh
> Priority: Major
>
> See [relevant email
> thread|https://lists.apache.org/thread.html/2e87f0adcdf8d42317765f298e3e6fdba72917a72d4a12e71e67e4b5@%3Cdev.beam.apache.org%3E].
> From [~echauchot]:
>
> _AFAIK Dataflow being a cloud hosted engine, the related runner is very
> different from the others. It just submits a job to the cloud hosted engine.
> So, no access to metrics container etc... from the runner. So I think that
> the MetricsPusher (component responsible for merging metrics and pushing them
> to a sink backend) must not be instanciated in DataflowRunner otherwise it
> would be more a client (driver) piece of code and we will lose all the
> interest of being close to the execution engine (among other things
> instrumentation of the execution of the pipelines). I think that the
> MetricsPusher needs to be instanciated in the actual Dataflow engine._
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
