Re: [PROPOSAL] Pipeline Runner API design doc

2016-08-02 Thread Kenneth Knowles
Hi, Yes, there are a few things "TODO" including aggregators and triggers. Triggers can either be an inline syntax tree or flattened and using "pointers" like the transforms and PCollections. With coders we've hit issues with the nesting and repetition that leads us to keep them flattened.

Re: [PROPOSAL] Pipeline Runner API design doc

2016-08-02 Thread Aljoscha Krettek
Hi, thanks for putting this together. Now that I'm seeing them side by side I think the Avro schema looks a lot nicer than the JSON schema but it's probably alright since we don't want to change this often (as you already said). The advantage of JSON is that the (intermediate) plans can easily be

Re: [PROPOSAL] Pipeline Runner API design doc

2016-07-14 Thread Kenneth Knowles
Hi everyone, I wanted to circle back on this thread and with another invitation to a discussion. Work on the high level refactorings to align the Java SDK with the primitives of the proposed model is pretty far along, as is moving out the stuff that we don't want in the user-facing SDK. Since

Re: [PROPOSAL] Pipeline Runner API design doc

2016-03-24 Thread Jean-Baptiste Onofré
Hi Kenn, thanks for the update. I'm reading it now. Regards JB On 03/23/2016 10:17 PM, Kenneth Knowles wrote: Hi everyone, Incorporating the feedback from the 1-pager I circulated a week ago, I have put together a concrete design document for the new API(s).

[PROPOSAL] Pipeline Runner API design doc

2016-03-23 Thread Kenneth Knowles
Hi everyone, Incorporating the feedback from the 1-pager I circulated a week ago, I have put together a concrete design document for the new API(s). https://docs.google.com/document/d/1bao-5B6uBuf-kwH1meenAuXXS0c9cBQ1B2J59I3FiyI/edit?usp=sharing I appreciate any and all feedback on the design.