Re: DataflowRunner | Cross-language

2020-06-08 Thread Ahmet Altay
On Mon, Jun 8, 2020 at 2:06 PM Chad Dombrova wrote: > Even when running portably, Dataflow still has its own implementation of >> PubSubIO that is switched out for Python's "implementation." (It's actually >> built into the same layer that provides the shuffle/group-by-key >> implementation.)

Re: DataflowRunner | Cross-language

2020-06-08 Thread Chad Dombrova
> Even when running portably, Dataflow still has its own implementation of > PubSubIO that is switched out for Python's "implementation." (It's actually > built into the same layer that provides the shuffle/group-by-key > implementation.) However, if you used the external Java PubSubIO it may not

Re: DataflowRunner | Cross-language

2020-06-08 Thread Robert Bradshaw
On Mon, Jun 8, 2020 at 12:57 PM Chad Dombrova wrote: > Hi all, > quick followup question: > > >> small correction. While the new runner will be available with Beam 2.21, >>> the Cross-Language support will be available in 2.22. >>> There will be limitations in the initial set of connectors you

Re: DataflowRunner | Cross-language

2020-06-08 Thread Chad Dombrova
Hi all, quick followup question: > small correction. While the new runner will be available with Beam 2.21, >> the Cross-Language support will be available in 2.22. >> There will be limitations in the initial set of connectors you can use >> with Cross-Lang. But at least you will have something

Re: DataflowRunner | Cross-language

2020-05-26 Thread Robert Bradshaw
On Tue, May 26, 2020 at 4:12 PM Sergei Sokolenko wrote: > small correction. While the new runner will be available with Beam 2.21, > the Cross-Language support will be available in 2.22. > There will be limitations in the initial set of connectors you can use > with Cross-Lang. But at least you

Re: DataflowRunner | Cross-language

2020-05-26 Thread Sergei Sokolenko
small correction. While the new runner will be available with Beam 2.21, the Cross-Language support will be available in 2.22. There will be limitations in the initial set of connectors you can use with Cross-Lang. But at least you will have something to test with, starting in 2.22 On Tue, May

Re: DataflowRunner | Cross-language

2020-05-26 Thread Sergei Sokolenko
More info will be forthcoming after Beam 2.21 is out. There will be a docs page describing how it all works. On Thu, May 21, 2020 at 11:18 PM Paweł Urbanowicz < pawel.urbanow...@polidea.com> wrote: > Hello, community, > > I found information that Google is working on supporting Dataflow runner >

Re: DataflowRunner | Cross-language

2020-05-26 Thread Chamikara Jayalath
We are working on making Kafka IO available to Python streaming users on Dataflow through cross-language transforms. There's no ETA for the availability of the framework in general for Dataflow yet. Thanks, Cham On Thu, May 21, 2020 at 11:18 PM Paweł Urbanowicz < pawel.urbanow...@polidea.com>

DataflowRunner | Cross-language

2020-05-22 Thread Paweł Urbanowicz
Hello, community, I found information that Google is working on supporting Dataflow runner for cross-language (https://beam.apache.org/roadmap/connectors-multi-sdk/) Is there any more information about the expected release of this feature? Thanks

DataflowRunner | Cross-language

2020-05-21 Thread Paweł Urbanowicz
Hello, community, I found information that Google is working on supporting Dataflow runner for cross-language (https://beam.apache.org/roadmap/connectors-multi-sdk/) Is there any more information about the expected release of this feature? Thanks