Hi Romain,
1. A runner has to implement the 5 (or 6 depending how we count) primitives.
It's the purpose of Runner API (the Fn API is more to deal with SDKs in the
runner). So, the Runner API is kind of adapter, and it implements all other
functions based on the primitives. You would like a formal RunnerAdapter in the
Runner API ?
2. A PipelineVisitor allows you to browse the DAG and get the PInput/POutput
(representing PCollection and side inputs/outputs) for each PTransforms/steps in
the pipeline. You can use this PipelineVisitor in a pre-step to get the complete
graph. I see in your code that you use it. What would you like instead ? Kind of
"object" representing the DAG ?
I gonna take a look on the code. FYI, I also have PoC for Hazelcast (with
Cellar), Ignite and MapReduce if you want to take a look.
Regards
JB
On 05/24/2017 10:54 PM, Romain Manni-Bucau wrote:
Hi guys,
congrats for the 2.0!
I have a few question regarding a custom runner implementation, there is no
particular order but adding numbers for later references:
1. why beam doesn't have (yet?) a RunnerAdapter with all primivite listed and
enforced by API, if I got right the code (shout if not) each runner is creating
its own processing context and convertion at the moment which kind of means beam
doesn't abstract the runtime which makes it harder to enter into beam model IMHO
(vs defining by contract all operations - potentially with defaults when
compositions are possible)
2. close to previous point (still around runner): i find quite hard to browse
the DAG of beam compared to a plain DAG, is it intended to be so low level,
can't we get a simple graph model?
Maybe I'm just hitting a not yet extended/defined land and therefore an user
friendly API is missing or I missed a central concept - in this case shout :p.
Any pointers would be very welcomed on how to implement a runner without
redefining a full transpiler/converter - or is it outside beam scope?
FYI here what I have https://github.com/rmannibucau/beam-hazelcast-runner and
here where I'm stucked (hesitating to redefine a full transpiler since I
expected beam to help):
https://github.com/rmannibucau/beam-hazelcast-runner/blob/master/src/main/java/com/github/rmannibucau/beam/runner/hazelcast/HazelcastPipelineVisitor.java
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://blog-rmannibucau.rhcloud.com> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
<https://javaeefactory-rmannibucau.rhcloud.com>
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
