Just curious here. What happens when this local JVM context dies for any reason? How does it work with DataFlow?
On Thu, Oct 4, 2018 at 1:50 AM Scott Wegner <[email protected]> wrote: > Another point that we discussed at ApacheCon is that a difference between > Dataflow and other runners is Dataflow is service-based and doesn't need a > locally executing "driver" program. A local driver context is a good place > to implement MetricsPusher because it is a singleton process. > > In fact, DataflowRunner supports PipelineResult.waitUntilFinish() [1], > where we do maintain the local JVM context. Currently in this mode the > runner polls the Dataflow service API for log messages [2]. It would be > very easy to also poll for metric updates and push them out via > MetricsPusher. > > [1] > https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java#L169 > [2] > https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java#L291 > > On Wed, Oct 3, 2018 at 4:44 AM Etienne Chauchot <[email protected]> > wrote: > >> Hi Scott, >> Thanks for the update. >> Both solutions look good to me. Though, they both have plus and minus. I >> let the googlers chose which is more appropriate: >> >> - DAG modifcation: less intrusive in Dataflow but the DAG executed and >> shown in the DAG UI in dataflow will contain an extra step that the user >> might wonder about. >> - polling thread: it is exactly what I did for the other runners, it is >> more transparent to the user but requires more infra work (adds a container >> that needs to be resilient) >> >> Best >> Etienne >> >> Le vendredi 21 septembre 2018 à 12:46 -0700, Scott Wegner a écrit : >> >> Hi Etienne, sorry for the delay on this. I just got back from leave and >> found this discussion. >> >> We haven't started implementing MetricsPusher in the Dataflow runner, >> mostly because the Dataflow service has it's own rich Metrics REST API and >> we haven't heard a need from Dataflow customers to push metrics to an >> external backend. However, it would be nice to have this implemented across >> all runners for feature parity. >> >> I read through the discussion in JIRA [1], and the simplest >> implementation for Dataflow may be to have a single thread periodically >> poll the Dataflow REST API [2] for latest metric values, and push them to a >> configured sink. This polling thread could be hosted in a separate docker >> container, within the worker process, or perhaps a ParDo with timers that >> gets injected into the pipeline during graph translation. >> >> At any rate, I'm not aware of anybody currently working on this. But with >> the Dataflow worker code being donated to Beam [3], soon it will be >> possible for anybody to contribute. >> >> [1] https://issues.apache.org/jira/browse/BEAM-3926 >> [2] >> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/getMetrics >> [3] >> https://lists.apache.org/thread.html/2bdc645659e2fbd7e29f3a2758941faefedb01148a2a11558dfe60f8@%3Cdev.beam.apache.org%3E >> >> On Fri, Aug 17, 2018 at 4:26 PM Lukasz Cwik <[email protected]> wrote: >> >> I forwarded your request to a few people who work on the internal parts >> of Dataflow to see if they could help in some way. >> >> On Thu, Aug 16, 2018 at 6:22 AM Etienne Chauchot <[email protected]> >> wrote: >> >> Hi all >> >> As we already discussed, it would be good to support Metrics Pusher [1] >> in Dataflow (in other runners also, of course). Today, only Spark and Flink >> support it. It requires a modification in C++ Dataflow code, so only Google >> friends can do it. >> >> Is someone interested in doing it ? >> >> Here is the ticket https://issues.apache.org/jira/browse/BEAM-3926 >> >> Besides, I wonder if this feature should be added to the capability >> matrix. >> >> [1] >> https://cwiki.apache.org/confluence/display/BEAM/Metrics+architecture+inside+the+runners >> >> Thanks >> Etienne >> >> >> >> > > -- > > > > > Got feedback? tinyurl.com/swegner-feedback >
