Re: Beam Fn API

2017-01-20 Thread Robert Bradshaw
Also, note that we can still support the "simple" case. For example, if the user supplies us with a jar file (as they do now) a runner could launch it as a subprocesses and communicate with it via this same Fn API or install it in a fixed container itself--the user doesn't *need* to know about

Re: Runner-provided ValueProviders

2017-01-20 Thread Kenneth Knowles
Note that the Runner API makes the problem and solution cross-runner, but the problem exists separately for each runner today. So a near-term solution makes sense as long as it is forward-looking (broadcasting runner-provided info to user fns) or has a TTL (putting extra stuff into BoundedSource).

Re: [VOTE] Merge Python SDK to the master branch

2017-01-20 Thread Dan Halperin
[X] +1, Merge python-sdk branch to master after the 0.5.0 release, and release it in the subsequent minor release. Thanks and woo! On Fri, Jan 20, 2017 at 12:00 PM, Jean-Baptiste Onofré wrote: > +1 to merge Python SDK after 0.5.0 release. > > Regards > JB > > > On 01/20/2017

Re: Runner-provided ValueProviders

2017-01-20 Thread Dan Halperin
I think this was initially motivated by BEAM-758 . Copying from that issue: In the forthcoming runner API, a user will be able to save a pipeline to JSON and then run it repeatedly. Many pieces of code (e.g., BigQueryIO.Read or Write) rely

Re: [DISCUSS] Python SDK status and next steps

2017-01-20 Thread Kenneth Knowles
To clarify the implied criteria of that last exchange, it is "An SDK should have at least one runner that can execute the complete model (may be a direct runner)" I want to highlight this, because whether an _SDK_ supports unbounded data is not particularly well-defined, and will evolve: - With

Re: Hosting data stores for IO Transform testing

2017-01-20 Thread Stephen Sisk
hey folks! I wanted to gather any last thoughts that people might have. I'd like to get started setting this up - anyone else have input? S On Thu, Jan 19, 2017 at 11:41 AM Stephen Sisk wrote: > Glad to hear you support kubernetes (although to be clear, I'm rooting for > the

Re: [VOTE] Merge Python SDK to the master branch

2017-01-20 Thread Davor Bonaci
[X] +1, Merge python-sdk branch to master after the 0.5.0 release, and release it in the subsequent minor release. On Fri, Jan 20, 2017 at 9:03 AM, Ahmet Altay wrote: > Hi all, > > > Please review the earlier discussion on the status of the Python SDK [1] > and vote

Re: [DISCUSS] Python SDK status and next steps

2017-01-20 Thread Ahmet Altay
Thank you Dan. Adding support for unbounded data is on the roadmap and it will be added to Python SDK soon. Thank you all again, I will start the official voting thread. Thank you, Ahmet On Thu, Jan 19, 2017 at 11:56 PM, Dan Halperin wrote: > I do not think that