+dev I read the document on Drain, and it sounds very promising. I have a few questions, starting with this statement from the doc:
"This document proposes a new pipeline action called Drain. Drain can be implemented by runners by manipulating the watermark of the pipeline." What is a pipeline "action" and how would this be exposed to the user? I assume this is externally and manually initiated. Is this something that would be invoked from a PipelineResult object (i.e. akin to "cancel")? Once a Drain is initiated on a pipeline, does this trigger a loop over all unbounded sources to set their watermark to infinity, or only certain ones? thanks, -chad On Fri, Aug 16, 2019 at 2:47 PM Jose Delgado <[email protected]> wrote: > I see, thank you Lukasz. > > > > Regards, > Jose > > *From: *Lukasz Cwik <[email protected]> > *Reply-To: *"[email protected]" <[email protected]> > *Date: *Monday, August 5, 2019 at 11:11 AM > *To: *user <[email protected]> > *Subject: *Re: Save state on tear down > > > > This is not possible today. > > > > There have been discussions about pipeline drain, snapshot and update [1, > 2] which may provide additional details of what is planned and could use > your feedback. > > > > 1: > https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8 > > 2: > https://docs.google.com/document/d/1UWhnYPgui0gUYOsuGcCjLuoOUlGA4QaY91n8p3wz9MY > > > > On Thu, Aug 1, 2019 at 3:44 PM Jose Delgado <[email protected]> > wrote: > > Hello All, > > > > I wondering if there is a way or pattern to save state on tear down ? > > > > In case of a failure or a pipeline we cannot update(due to significant > changes ) we would like to save the state and re-load it on the next > creation of the pipeline. > > > > Note: we are currently using Google Dataflow runner > > > > Regards, > > Jose > >
