Re: Need advice: PubsubIO external transform PR

> The issue is also tracked here: > https://jira.apache.org/jira/browse/BEAM-7870 There are some suggestions > in the issue. I think the best solution is to allow execution of the > source API parts of KafkaIO/PubSubIO (on the Runner) and the following > UDFs (in the environment). Since those do

Re: Need advice: PubsubIO external transform PR

On Tue, Aug 20, 2019 at 4:29 AM Maximilian Michels wrote: > Hi Chad! > > Thank you so much for your feedback. You are 100% on the right track. > What you are seeing is a core issue that also needs to be solved for > KafkaIO to be fully usable in other SDKs. I haven't had much time to > work on

Re: Need advice: PubsubIO external transform PR

Hi Chad! Thank you so much for your feedback. You are 100% on the right track. What you are seeing is a core issue that also needs to be solved for KafkaIO to be fully usable in other SDKs. I haven't had much time to work on this in the past weeks but now is the time :) The cross-language

Re: Need advice: PubsubIO external transform PR

> I don't understand why this replacement is necessary, since the next >>> transform in the chain is a java ParDo that seems like it should be fully >>> capable of using PubsubMessageWithAttributesCoder. >>> >> > Not too familiar with Flink, but have you tried using PubSub source from a > pure

Re: Need advice: PubsubIO external transform PR

On Mon, Aug 19, 2019 at 5:48 PM Ahmet Altay wrote: > Adding a few folks who might help +Chamikara Jayalath > +Lukasz Cwik +Maximilian Michels > > > On Sat, Aug 17, 2019 at 11:27 AM Chad Dombrova wrote: > >> Hi all, >> I've got a PR[1] for adding >>

Re: Need advice: PubsubIO external transform PR

Adding a few folks who might help +Chamikara Jayalath +Lukasz Cwik +Maximilian Michels On Sat, Aug 17, 2019 at 11:27 AM Chad Dombrova wrote: > Hi all, > I've got a PR[1] for adding > external transform support to PubsubIO so that it will work with

Need advice: PubsubIO external transform PR

Hi all, I've got a PR[1] for adding external transform support to PubsubIO so that it will work with python and go pipelines on Flink, and I am *so* close, but I've run into questions that the code cannot answer: I need a human now. The brief summary is