IF the local JVM dies, the job keeps running. On Thu, Oct 4, 2018 at 12:03 AM Jozef Vilcek <[email protected]> wrote:
> Just curious here. What happens when this local JVM context dies for any > reason? How does it work with DataFlow? > > On Thu, Oct 4, 2018 at 1:50 AM Scott Wegner <[email protected]> wrote: > >> Another point that we discussed at ApacheCon is that a difference between >> Dataflow and other runners is Dataflow is service-based and doesn't need a >> locally executing "driver" program. A local driver context is a good place >> to implement MetricsPusher because it is a singleton process. >> >> In fact, DataflowRunner supports PipelineResult.waitUntilFinish() [1], >> where we do maintain the local JVM context. Currently in this mode the >> runner polls the Dataflow service API for log messages [2]. It would be >> very easy to also poll for metric updates and push them out via >> MetricsPusher. >> >> [1] >> https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java#L169 >> [2] >> https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java#L291 >> >> On Wed, Oct 3, 2018 at 4:44 AM Etienne Chauchot <[email protected]> >> wrote: >> >>> Hi Scott, >>> Thanks for the update. >>> Both solutions look good to me. Though, they both have plus and minus. I >>> let the googlers chose which is more appropriate: >>> >>> - DAG modifcation: less intrusive in Dataflow but the DAG executed and >>> shown in the DAG UI in dataflow will contain an extra step that the user >>> might wonder about. >>> - polling thread: it is exactly what I did for the other runners, it is >>> more transparent to the user but requires more infra work (adds a container >>> that needs to be resilient) >>> >>> Best >>> Etienne >>> >>> Le vendredi 21 septembre 2018 à 12:46 -0700, Scott Wegner a écrit : >>> >>> Hi Etienne, sorry for the delay on this. I just got back from leave and >>> found this discussion. >>> >>> We haven't started implementing MetricsPusher in the Dataflow runner, >>> mostly because the Dataflow service has it's own rich Metrics REST API and >>> we haven't heard a need from Dataflow customers to push metrics to an >>> external backend. However, it would be nice to have this implemented across >>> all runners for feature parity. >>> >>> I read through the discussion in JIRA [1], and the simplest >>> implementation for Dataflow may be to have a single thread periodically >>> poll the Dataflow REST API [2] for latest metric values, and push them to a >>> configured sink. This polling thread could be hosted in a separate docker >>> container, within the worker process, or perhaps a ParDo with timers that >>> gets injected into the pipeline during graph translation. >>> >>> At any rate, I'm not aware of anybody currently working on this. But >>> with the Dataflow worker code being donated to Beam [3], soon it will be >>> possible for anybody to contribute. >>> >>> [1] https://issues.apache.org/jira/browse/BEAM-3926 >>> [2] >>> https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/getMetrics >>> [3] >>> https://lists.apache.org/thread.html/2bdc645659e2fbd7e29f3a2758941faefedb01148a2a11558dfe60f8@%3Cdev.beam.apache.org%3E >>> >>> On Fri, Aug 17, 2018 at 4:26 PM Lukasz Cwik <[email protected]> wrote: >>> >>> I forwarded your request to a few people who work on the internal parts >>> of Dataflow to see if they could help in some way. >>> >>> On Thu, Aug 16, 2018 at 6:22 AM Etienne Chauchot <[email protected]> >>> wrote: >>> >>> Hi all >>> >>> As we already discussed, it would be good to support Metrics Pusher [1] >>> in Dataflow (in other runners also, of course). Today, only Spark and Flink >>> support it. It requires a modification in C++ Dataflow code, so only Google >>> friends can do it. >>> >>> Is someone interested in doing it ? >>> >>> Here is the ticket https://issues.apache.org/jira/browse/BEAM-3926 >>> >>> Besides, I wonder if this feature should be added to the capability >>> matrix. >>> >>> [1] >>> https://cwiki.apache.org/confluence/display/BEAM/Metrics+architecture+inside+the+runners >>> >>> Thanks >>> Etienne >>> >>> >>> >>> >> >> -- >> >> >> >> >> Got feedback? tinyurl.com/swegner-feedback >> >
