Hi Romain, These are great topics for discussion, and very good timing.
I strongly recommend joining dev@beam, as that is where most discussion about implementing Beam runners takes place. On that list, I recently shared a guide for implementing a Beam runner at https://s.apache.org/beam-runner-guide. It will be improved and appear on the Beam site fairly soon. I have also taken the liberty of moving this discussion to dev@beam and moved user@beam to BCC. Kenn On Wed, May 24, 2017 at 1:54 PM, Romain Manni-Bucau <[email protected]> wrote: > Hi guys, > > congrats for the 2.0! > > I have a few question regarding a custom runner implementation, there is > no particular order but adding numbers for later references: > > 1. why beam doesn't have (yet?) a RunnerAdapter with all primivite listed > and enforced by API, if I got right the code (shout if not) each runner is > creating its own processing context and convertion at the moment which kind > of means beam doesn't abstract the runtime which makes it harder to enter > into beam model IMHO (vs defining by contract all operations - potentially > with defaults when compositions are possible) > 2. close to previous point (still around runner): i find quite hard to > browse the DAG of beam compared to a plain DAG, is it intended to be so low > level, can't we get a simple graph model? > > Maybe I'm just hitting a not yet extended/defined land and therefore an > user friendly API is missing or I missed a central concept - in this case > shout :p. > > Any pointers would be very welcomed on how to implement a runner without > redefining a full transpiler/converter - or is it outside beam scope? > > > FYI here what I have https://github.com/rmannibucau/beam-hazelcast-runner > and here where I'm stucked (hesitating to redefine a full transpiler since > I expected beam to help): https://github.com/rmannibucau/beam-hazelcast- > runner/blob/master/src/main/java/com/github/rmannibucau/ > beam/runner/hazelcast/HazelcastPipelineVisitor.java > > Romain Manni-Bucau > @rmannibucau <https://twitter.com/rmannibucau> | Blog > <https://blog-rmannibucau.rhcloud.com> | Old Blog > <http://rmannibucau.wordpress.com> | Github > <https://github.com/rmannibucau> | LinkedIn > <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory > <https://javaeefactory-rmannibucau.rhcloud.com> >
