I believe so, but it looks like the dispenser is an interface to the user for such features. We still need to define what the semantics of features like Drain are, and how they affect the pipeline.
On Tue, Jun 6, 2017 at 12:06 PM, Jean-Baptiste Onofré <[email protected]> wrote: > Hi Reuven, > > In the "Apache Beam: Technical vision" document (dating from the > incubation) (https://docs.google.com/document/d/1UyAeugHxZmVlQ5cEWo_eOPg > XNQA1oD-rGooWOSwAqh8/edit?usp=sharing), I added a section named "Beam > Pipelines Dispenser". > > The idea is to be able to bootstrap, run and control pipelines (and the > runners). > > I think it's somehow related. WDYT ? > > Regards > JB > > On 06/06/2017 07:43 PM, Reuven Lax wrote: > >> Hi all, >> >> Beam is a great programming mode, but in order to really run pipelines >> (especially streaming pipelines which are "always on") in a production >> setting, there is a set of features necessary. Dataflow has a couple of >> those features built in (Drain and Update), and inspired by those I'll be >> sending out a few proposals for similar features in Beam. >> >> Please note that my intention here is _not_ to simply forklift the >> Dataflow >> features to Beam. The Dataflow features are being used as inspiration, and >> we have two years of experience how real users have used these feature >> (and >> also experienced when users have found these features limited and >> frustrating). In every case my Beam proposals are different - hopefully >> better! - than the actual Dataflow feature that exists today. >> >> I think all runners would greatly benefit from production-control features >> like this, and I would love to see community input. The first proposal is >> for a way of draining a streaming pipeline before stopping it, and here it >> is >> <https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8X >> GhDPmm3cllSN8IMmWci8/edit> >> . >> >> Reuven >> >> > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
