What is the update / compat story around schemas? * are unknown fields propagated through if the user only reads/modifies a row? * how does it work in a pipeline update scenario (downgrade / upgrade)?
Boyuan has been working on a Kafka via SDF source and have been trying to figure out which interchange format to use for the "source descriptors" that feed into the SDF. Some obvious choices are json, avro, proto, and Beam schemas all with their caveats. On Fri, Jun 12, 2020 at 1:32 PM Brian Hulette <bhule...@google.com> wrote: > Thanks! I see there are jiras for SpannerIO and JdbcIO as part of that. > Are you planning on using row coder for them? > If so I want to make sure you're aware of > https://s.apache.org/beam-schema-io (sent to the dev list last week [1]). > +Scott > Lukas <slu...@google.com> will be working on building out the ideas there > this summer. His work could be useful for making these IOs cross-language > (and you would get a mapping to SQL out of it without much more effort). > > Brian > > [1] > https://lists.apache.org/thread.html/rc1695025d41c5dc38cdf7bc32bea0e7421379b1c543c2d82f69aa179%40%3Cdev.beam.apache.org%3E > > On Tue, Jun 2, 2020 at 9:30 AM Piotr Szuberski < > piotr.szuber...@polidea.com> wrote: > >> Sure, I'll do that >> >> On 2020/05/28 17:54:49, Chamikara Jayalath <chamik...@google.com> wrote: >> > Great. Thanks for working on this. Can you please add these tasks and >> JIRAs >> > to the cross-language transforms roadmap under "Connector/transform >> > support". >> > https://beam.apache.org/roadmap/connectors-multi-sdk/ >> > >> > Happy to help if you run into any issues during this task. >> > >> > <https://beam.apache.org/roadmap/connectors-multi-sdk/>Thanks, >> > Cham >> > >> > On Thu, May 28, 2020 at 9:59 AM Piotr Szuberski < >> piotr.szuber...@polidea.com> >> > wrote: >> > >> > > I added to Jira task of creating cross-language wrappers for Java >> IOs. It >> > > will soon be in progress. >> > > >> > >> >
