+Boyuan Zhang <boyu...@google.com> On Fri, Jun 12, 2020 at 3:32 PM Luke Cwik <lc...@google.com> wrote:
> What is the update / compat story around schemas? > * are unknown fields propagated through if the user only reads/modifies a > row? > * how does it work in a pipeline update scenario (downgrade / upgrade)? > > Boyuan has been working on a Kafka via SDF source and have been trying to > figure out which interchange format to use for the "source descriptors" > that feed into the SDF. Some obvious choices are json, avro, proto, and > Beam schemas all with their caveats. > > On Fri, Jun 12, 2020 at 1:32 PM Brian Hulette <bhule...@google.com> wrote: > >> Thanks! I see there are jiras for SpannerIO and JdbcIO as part of that. >> Are you planning on using row coder for them? >> If so I want to make sure you're aware of >> https://s.apache.org/beam-schema-io (sent to the dev list last week >> [1]). +Scott Lukas <slu...@google.com> will be working on building out >> the ideas there this summer. His work could be useful for making these IOs >> cross-language (and you would get a mapping to SQL out of it without much >> more effort). >> >> Brian >> >> [1] >> https://lists.apache.org/thread.html/rc1695025d41c5dc38cdf7bc32bea0e7421379b1c543c2d82f69aa179%40%3Cdev.beam.apache.org%3E >> >> On Tue, Jun 2, 2020 at 9:30 AM Piotr Szuberski < >> piotr.szuber...@polidea.com> wrote: >> >>> Sure, I'll do that >>> >>> On 2020/05/28 17:54:49, Chamikara Jayalath <chamik...@google.com> >>> wrote: >>> > Great. Thanks for working on this. Can you please add these tasks and >>> JIRAs >>> > to the cross-language transforms roadmap under "Connector/transform >>> > support". >>> > https://beam.apache.org/roadmap/connectors-multi-sdk/ >>> > >>> > Happy to help if you run into any issues during this task. >>> > >>> > <https://beam.apache.org/roadmap/connectors-multi-sdk/>Thanks, >>> > Cham >>> > >>> > On Thu, May 28, 2020 at 9:59 AM Piotr Szuberski < >>> piotr.szuber...@polidea.com> >>> > wrote: >>> > >>> > > I added to Jira task of creating cross-language wrappers for Java >>> IOs. It >>> > > will soon be in progress. >>> > > >>> > >>> >>
