Re: The state of external transforms in Beam

2019-11-08 Thread Chamikara Jayalath
Send https://github.com/apache/beam/pull/10054 to update the roadmap. Thanks, Cham On Mon, Nov 4, 2019 at 10:24 AM Chamikara Jayalath wrote: > Makes sense. > > I can look into expanding on what we have at following location and adding > links to some of the existing work as a first step. >

Re: The state of external transforms in Beam

2019-11-04 Thread Chamikara Jayalath
Makes sense. I can look into expanding on what we have at following location and adding links to some of the existing work as a first step. https://beam.apache.org/roadmap/connectors-multi-sdk/ Created https://issues.apache.org/jira/browse/BEAM-8553 We also need more detailed documentation for

Re: The state of external transforms in Beam

2019-11-03 Thread Thomas Weise
This thread was very helpful to find more detail in https://jira.apache.org/jira/browse/BEAM-7870 It would be great to have cross-language current state mentioned as top level entry on https://beam.apache.org/roadmap/ On Mon, Sep 16, 2019 at 6:07 PM Chamikara Jayalath wrote: > Thanks for the

Re: The state of external transforms in Beam

2019-09-16 Thread Chamikara Jayalath
Thanks for the nice write up Chad. On Mon, Sep 16, 2019 at 12:17 PM Robert Bradshaw wrote: > Thanks for bringing this up again. My thoughts on the open questions below. > > On Mon, Sep 16, 2019 at 11:51 AM Chad Dombrova wrote: > > That commit solves 2 problems: > > > > Adds the pubsub Java

Re: The state of external transforms in Beam

2019-09-16 Thread Robert Bradshaw
Thanks for bringing this up again. My thoughts on the open questions below. On Mon, Sep 16, 2019 at 11:51 AM Chad Dombrova wrote: > That commit solves 2 problems: > > Adds the pubsub Java deps so that they’re available in our portable pipeline > Makes the coder for the PubsubIO message-holder

The state of external transforms in Beam

2019-09-16 Thread Chad Dombrova
Hi all, There was some interest in this topic at the Beam Summit this week (btw, great job to everyone involved!), so I thought I’d try to summarize the current state of things. First, let me explain the idea behind an external transforms for the uninitiated. Problem: - there’s a transform