2017-05-25 7:42 GMT+02:00 Jean-Baptiste Onofré <[email protected]>:
> Hi Romain,
>
> 1. A runner has to implement the 5 (or 6 depending how we count)
> primitives. It's the purpose of Runner API (the Fn API is more to deal with
> SDKs in the runner). So, the Runner API is kind of adapter, and it
> implements all other functions based on the primitives. You would like a
> formal RunnerAdapter in the Runner API ?
>
Yes, or at least change the communication in conferences ;) ("have to
implement" vs "have to handle").
>
> 2. A PipelineVisitor allows you to browse the DAG and get the
> PInput/POutput (representing PCollection and side inputs/outputs) for each
> PTransforms/steps in the pipeline. You can use this PipelineVisitor in a
> pre-step to get the complete graph. I see in your code that you use it.
> What would you like instead ? Kind of "object" representing the DAG ?
>
There I was wondering why the visitor was not more typed:
void onParDoFn(in, out)
void onGroupByKey(in, out)
etc...
and probably in a base class the helpers we have in the core construction
(SimpleDoFn) to make the implementation as smooth as possible
>
> I gonna take a look on the code. FYI, I also have PoC for Hazelcast (with
> Cellar), Ignite and MapReduce if you want to take a look.
>
Sure! Think Jet makes sense since it is the "official" hazelcast data
processing solution but shouldn't be that far :).
>
> Regards
> JB
>
> On 05/24/2017 10:54 PM, Romain Manni-Bucau wrote:
>
>> Hi guys,
>>
>> congrats for the 2.0!
>>
>> I have a few question regarding a custom runner implementation, there is
>> no particular order but adding numbers for later references:
>>
>> 1. why beam doesn't have (yet?) a RunnerAdapter with all primivite listed
>> and enforced by API, if I got right the code (shout if not) each runner is
>> creating its own processing context and convertion at the moment which kind
>> of means beam doesn't abstract the runtime which makes it harder to enter
>> into beam model IMHO (vs defining by contract all operations - potentially
>> with defaults when compositions are possible)
>> 2. close to previous point (still around runner): i find quite hard to
>> browse the DAG of beam compared to a plain DAG, is it intended to be so low
>> level, can't we get a simple graph model?
>>
>> Maybe I'm just hitting a not yet extended/defined land and therefore an
>> user friendly API is missing or I missed a central concept - in this case
>> shout :p.
>>
>> Any pointers would be very welcomed on how to implement a runner without
>> redefining a full transpiler/converter - or is it outside beam scope?
>>
>>
>> FYI here what I have https://github.com/rmannibucau/beam-hazelcast-runner
>> and here where I'm stucked (hesitating to redefine a full transpiler since
>> I expected beam to help): https://github.com/rmannibucau
>> /beam-hazelcast-runner/blob/master/src/main/java/com/
>> github/rmannibucau/beam/runner/hazelcast/HazelcastPipelineVisitor.java
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog <
>> https://blog-rmannibucau.rhcloud.com> | Old Blog <
>> http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibuca
>> u> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | JavaEE Factory
>> <https://javaeefactory-rmannibucau.rhcloud.com>
>>
>
> --
> Jean-Baptiste Onofré
> [email protected]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
