Hi Etienne, sorry for the delay on this. I just got back from leave and found this discussion.
We haven't started implementing MetricsPusher in the Dataflow runner, mostly because the Dataflow service has it's own rich Metrics REST API and we haven't heard a need from Dataflow customers to push metrics to an external backend. However, it would be nice to have this implemented across all runners for feature parity. I read through the discussion in JIRA [1], and the simplest implementation for Dataflow may be to have a single thread periodically poll the Dataflow REST API [2] for latest metric values, and push them to a configured sink. This polling thread could be hosted in a separate docker container, within the worker process, or perhaps a ParDo with timers that gets injected into the pipeline during graph translation. At any rate, I'm not aware of anybody currently working on this. But with the Dataflow worker code being donated to Beam [3], soon it will be possible for anybody to contribute. [1] https://issues.apache.org/jira/browse/BEAM-3926 [2] https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/getMetrics [3] https://lists.apache.org/thread.html/2bdc645659e2fbd7e29f3a2758941faefedb01148a2a11558dfe60f8@%3Cdev.beam.apache.org%3E On Fri, Aug 17, 2018 at 4:26 PM Lukasz Cwik <[email protected]> wrote: > I forwarded your request to a few people who work on the internal parts of > Dataflow to see if they could help in some way. > > On Thu, Aug 16, 2018 at 6:22 AM Etienne Chauchot <[email protected]> > wrote: > >> Hi all >> >> As we already discussed, it would be good to support Metrics Pusher [1] >> in Dataflow (in other runners also, of course). Today, only Spark and Flink >> support it. It requires a modification in C++ Dataflow code, so only Google >> friends can do it. >> >> Is someone interested in doing it ? >> >> Here is the ticket https://issues.apache.org/jira/browse/BEAM-3926 >> >> Besides, I wonder if this feature should be added to the capability >> matrix. >> >> [1] >> https://cwiki.apache.org/confluence/display/BEAM/Metrics+architecture+inside+the+runners >> >> Thanks >> Etienne >> > -- Got feedback? tinyurl.com/swegner-feedback
