If you want to send across a "rich" data record, consider defining a schema
and using a row coder since row coder is XLang compatible.

On Thu, Jul 16, 2020 at 9:28 AM Robert Bradshaw <rober...@google.com> wrote:

> Note also that once you get the Bytes in Python, you can use whatever
> coder (or Map) to decode them that you want.
>
> On Thu, Jul 16, 2020 at 9:21 AM Boyuan Zhang <boyu...@google.com> wrote:
> >
> > Hi Piotr,
> >
> > X-Lang uses TypedWithoutMetadata, which outputs the KV directly instead
> of KafkaRecord: see here. Given the limit that x-lang can only work with
> well-known coders, if you want to process the KV in python output from
> KafkaIO, the coders of key and value should be well-known in beam. By
> default, the key and value are bytes: see here.
> >
> > On Thu, Jul 16, 2020 at 8:48 AM Piotr Szuberski <
> piotr.szuber...@polidea.com> wrote:
> >>
> >> I'm writing a python wrappers for KinesisIO and I encountered a problem
> that Read transform creates a PCollection with KinesisRecord class which's
> coder by default is assigned as 'beam:coders:javasdk:0.1'. I managed to
> register this coder using CoderTranslatorRegistrar which adds the coder to
> the KNOWN_CODER_URNS and therefore is sent with my custom urn.
> >>
> >> Kafka's cross language Write transform uses KV<>, is  encoded by
> default in beam.
> >>
> >> But I can't see how KafkaRecordCoder is translated in cross-language
> usage to python? I can't see any place in code where it gets registered.
> >>
> >> I just don't get how KafkaIO.Read works in cross-language. Could
> someone clarify me how does it work?
> >>
> >> Thanks in advance!
>

Reply via email to