Thanks Cham. Standard coder is a good point. Does it mean non-standard coder doesn't work when crossing language boundaries even if it is implemented in both Java and Python sdk?
On Mon, Jun 15, 2020 at 10:08 AM Chamikara Jayalath <[email protected]> wrote: > Thanks. +1 for using RowCoder. We should try to use standard coders [1] in > the x-lang SDK boundaries. > If we use other coders (for example, ProtoCoder) it may or may not work > depending on how various runners implement support for x-lang. > > This might require slightly updating existing transforms or adding > additional conversion transforms to cross-language builders. > > [1] > https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L669 > > Thanks, > Cham > > On Mon, Jun 15, 2020 at 10:00 AM Piotr Szuberski < > [email protected]> wrote: > >> Right now I'm working on JdbcIO and I'm using Row and Schema protobuffs. >> I'm figuring out how to use them properly. Thanks for the article - for >> sure it will be helpful! >> >> On 2020/06/12 20:32:16, Brian Hulette <[email protected]> wrote: >> > Thanks! I see there are jiras for SpannerIO and JdbcIO as part of that. >> Are >> > you planning on using row coder for them? >> > If so I want to make sure you're aware of >> > https://s.apache.org/beam-schema-io (sent to the dev list last week >> > [1]). +Scott >> > Lukas <[email protected]> will be working on building out the ideas >> there >> > this summer. His work could be useful for making these IOs >> cross-language >> > (and you would get a mapping to SQL out of it without much more effort). >> > >> > Brian >> > >> > [1] >> > >> https://lists.apache.org/thread.html/rc1695025d41c5dc38cdf7bc32bea0e7421379b1c543c2d82f69aa179%40%3Cdev.beam.apache.org%3E >> > >> > On Tue, Jun 2, 2020 at 9:30 AM Piotr Szuberski < >> [email protected]> >> > wrote: >> > >> > > Sure, I'll do that >> > > >> > > On 2020/05/28 17:54:49, Chamikara Jayalath <[email protected]> >> wrote: >> > > > Great. Thanks for working on this. Can you please add these tasks >> and >> > > JIRAs >> > > > to the cross-language transforms roadmap under "Connector/transform >> > > > support". >> > > > https://beam.apache.org/roadmap/connectors-multi-sdk/ >> > > > >> > > > Happy to help if you run into any issues during this task. >> > > > >> > > > <https://beam.apache.org/roadmap/connectors-multi-sdk/>Thanks, >> > > > Cham >> > > > >> > > > On Thu, May 28, 2020 at 9:59 AM Piotr Szuberski < >> > > [email protected]> >> > > > wrote: >> > > > >> > > > > I added to Jira task of creating cross-language wrappers for Java >> IOs. >> > > It >> > > > > will soon be in progress. >> > > > > >> > > > >> > > >> > >> >
