To clarify my previous point, I think transform
KafkaIO.Read.TypedWithoutMetadata [1] which produces a KV (for
example KV if we use ByteArraySerializer for keys and values)
should work in the current form if we don't have a runner specific override
for the source (hence allowing source and the
Ah, yes, registering a RowCoder seems like a fine solution here.
(Either that or have a wrapping PTransform that explicitly converts to
Rows or similar.)
On Wed, Feb 19, 2020 at 10:03 PM Chad Dombrova wrote:
>
> The Java deps are only half of the problem. The other half is that PubsubIO
> and
The Java deps are only half of the problem. The other half is that PubsubIO
and KafkaIO are using classes that do not have a python equivalent and thus
no universal coder. The solution discussed in the issue I linked above was
to use row coder registries in Java, to convert from these types to
Hopefully this should be resovled by
https://issues.apache.org/jira/browse/BEAM-9229
On Wed, Feb 19, 2020 at 5:52 PM Chad Dombrova wrote:
>
> We are using external transforms to get access to PubSubIO within python. It
> works well, but there is one major issue remaining to fix: we have to
On Wed, Feb 19, 2020 at 5:52 PM Chad Dombrova wrote:
> We are using external transforms to get access to PubSubIO within python.
> It works well, but there is one major issue remaining to fix: we have to
> build a custom beam with a hack to add the PubSubIO java deps and fix up
> the coders.
We are using external transforms to get access to PubSubIO within python.
It works well, but there is one major issue remaining to fix: we have to
build a custom beam with a hack to add the PubSubIO java deps and fix up
the coders. This affects KafkaIO as well. There's an issue here:
On Wed, Feb 12, 2020 at 8:10 AM Alexey Romanenko
wrote:
>
> AFAIK, there's no official guide for cross-language pipelines. But there
>> are examples and test cases you can use as reference such as:
>>
>>
> AFAIK, there's no official guide for cross-language pipelines. But there are
> examples and test cases you can use as reference such as:
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py
>
>
Thank you for response!
> AFAIK, there's no official guide for cross-language pipelines. But there are
> examples and test cases you can use as reference such as:
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_xlang.py
>
>
On Tue, Feb 11, 2020 at 11:13 AM Heejong Lee wrote:
>
>
> On Tue, Feb 11, 2020 at 9:37 AM Alexey Romanenko
> wrote:
>
>> Hi all,
>>
>> I just wanted to ask for more details about the status of cross-language
>> pipelines (rather, transforms). I see some discussions about that here, but
>> I
On Tue, Feb 11, 2020 at 9:37 AM Alexey Romanenko
wrote:
> Hi all,
>
> I just wanted to ask for more details about the status of cross-language
> pipelines (rather, transforms). I see some discussions about that here, but
> I think it’s more around cross-language IOs.
>
> I’ll appreciate for any
Hi all,
I just wanted to ask for more details about the status of cross-language
pipelines (rather, transforms). I see some discussions about that here, but I
think it’s more around cross-language IOs.
I’ll appreciate for any information about that topic and answers for these
questions:
- Are
12 matches
Mail list logo
