Hi Reuven,
In the "Apache Beam: Technical vision" document (dating from the incubation)
(https://docs.google.com/document/d/1UyAeugHxZmVlQ5cEWo_eOPgXNQA1oD-rGooWOSwAqh8/edit?usp=sharing),
I added a section named "Beam Pipelines Dispenser".
The idea is to be able to bootstrap, run and control pipelines (and the
runners).
I think it's somehow related. WDYT ?
Regards
JB
On 06/06/2017 07:43 PM, Reuven Lax wrote:
Hi all,
Beam is a great programming mode, but in order to really run pipelines
(especially streaming pipelines which are "always on") in a production
setting, there is a set of features necessary. Dataflow has a couple of
those features built in (Drain and Update), and inspired by those I'll be
sending out a few proposals for similar features in Beam.
Please note that my intention here is _not_ to simply forklift the Dataflow
features to Beam. The Dataflow features are being used as inspiration, and
we have two years of experience how real users have used these feature (and
also experienced when users have found these features limited and
frustrating). In every case my Beam proposals are different - hopefully
better! - than the actual Dataflow feature that exists today.
I think all runners would greatly benefit from production-control features
like this, and I would love to see community input. The first proposal is
for a way of draining a streaming pipeline before stopping it, and here it
is
<https://docs.google.com/document/d/1NExwHlj-2q2WUGhSO4jTu8XGhDPmm3cllSN8IMmWci8/edit>
.
Reuven
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
