On Tue, Jun 16, 2020 at 1:42 PM Piotr Szuberski <piotr.szuber...@polidea.com> wrote:
> I published a PR with Jdbc xlang write transform with python wrapper > (#12023, #12022) - could you suggest someone to assign for CR? I could help review these > > Btw. For now the standard row coder in python is very limited - it can > only use simple types (no bytes, datetime, boolean) but I suppose it'll be > developed in the near future? > BEAM-7996 is tracking support for the remaining primitive types in Python's row coder. None of these should be that difficult to add I just haven't had the time myself. Boolean should be especially quick since Chad made it a standard coder a while ago. DATETIME is harder. We want it to be a logical type rather than a primitive type for portable schemas. I was trying to do that with the MillisInstant PR [1], but that raised a question about logical types and SQL which we still haven't resolved [2]. [1] https://github.com/apache/beam/pull/11456 [2] https://lists.apache.org/thread.html/r2e05355b74fb5b8149af78ade1e3539ec08371a9a4b2b9e45737e6be%40%3Cdev.beam.apache.org%3E On 2020/06/15 17:08:29, Chamikara Jayalath <chamik...@google.com> wrote: > > Thanks. +1 for using RowCoder. We should try to use standard coders [1] > in > > the x-lang SDK boundaries. > > If we use other coders (for example, ProtoCoder) it may or may not work > > depending on how various runners implement support for x-lang. > > > > This might require slightly updating existing transforms or adding > > additional conversion transforms to cross-language builders. > > > > [1] > > > https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L669 > > > > Thanks, > > Cham > > > > On Mon, Jun 15, 2020 at 10:00 AM Piotr Szuberski < > > piotr.szuber...@polidea.com> wrote: > > > > > Right now I'm working on JdbcIO and I'm using Row and Schema > protobuffs. > > > I'm figuring out how to use them properly. Thanks for the article - for > > > sure it will be helpful! > > > > > > On 2020/06/12 20:32:16, Brian Hulette <bhule...@google.com> wrote: > > > > Thanks! I see there are jiras for SpannerIO and JdbcIO as part of > that. > > > Are > > > > you planning on using row coder for them? > > > > If so I want to make sure you're aware of > > > > https://s.apache.org/beam-schema-io (sent to the dev list last week > > > > [1]). +Scott > > > > Lukas <slu...@google.com> will be working on building out the ideas > > > there > > > > this summer. His work could be useful for making these IOs > cross-language > > > > (and you would get a mapping to SQL out of it without much more > effort). > > > > > > > > Brian > > > > > > > > [1] > > > > > > > > https://lists.apache.org/thread.html/rc1695025d41c5dc38cdf7bc32bea0e7421379b1c543c2d82f69aa179%40%3Cdev.beam.apache.org%3E > > > > > > > > On Tue, Jun 2, 2020 at 9:30 AM Piotr Szuberski < > > > piotr.szuber...@polidea.com> > > > > wrote: > > > > > > > > > Sure, I'll do that > > > > > > > > > > On 2020/05/28 17:54:49, Chamikara Jayalath <chamik...@google.com> > > > wrote: > > > > > > Great. Thanks for working on this. Can you please add these > tasks and > > > > > JIRAs > > > > > > to the cross-language transforms roadmap under > "Connector/transform > > > > > > support". > > > > > > https://beam.apache.org/roadmap/connectors-multi-sdk/ > > > > > > > > > > > > Happy to help if you run into any issues during this task. > > > > > > > > > > > > <https://beam.apache.org/roadmap/connectors-multi-sdk/>Thanks, > > > > > > Cham > > > > > > > > > > > > On Thu, May 28, 2020 at 9:59 AM Piotr Szuberski < > > > > > piotr.szuber...@polidea.com> > > > > > > wrote: > > > > > > > > > > > > > I added to Jira task of creating cross-language wrappers for > Java > > > IOs. > > > > > It > > > > > > > will soon be in progress. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
