[
https://issues.apache.org/jira/browse/BEAM-7029?focusedWorklogId=225662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-225662
]
ASF GitHub Bot logged work on BEAM-7029:
----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Apr/19 16:12
Start Date: 10/Apr/19 16:12
Worklog Time Spent: 10m
Work Description: lukecwik commented on issue #8251: [BEAM-7029] Add
KafkaIO.Read as external transform
URL: https://github.com/apache/beam/pull/8251#issuecomment-481756803
@mxm Supporting the whole KafkaRecord could be a TODO for the future. For
now you could just produce KV<byte[], byte[]> with the intent that the
receiving end performs any Kafka key/value -> language specific type decoding
(instead of passing in the key / value deserializers to the Java SDK). This
will produce a version that has a lot of value for many people since all "data"
types can be supported in the Python SDK by exposing a UDF that users can
implement to perform the decoding in the Python SDK. You could produce some
defaults for things like bytes/utf8/dates/...
Also, to support the whole KafkaRecord in the future, Kafka must already
have a well defined "encoding" format for compatibility across all its client
SDKs so it would be great if we could use that and then use the parsing in the
Kafka client libraries to do that work. Our coder spec would be
beam:coder:kafka_record:v1 and just point to a certain version of the
KafkaRecord spec.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 225662)
Time Spent: 7h (was: 6h 50m)
> Support KafkaIO to be configured externally for use with other SDKs
> -------------------------------------------------------------------
>
> Key: BEAM-7029
> URL: https://issues.apache.org/jira/browse/BEAM-7029
> Project: Beam
> Issue Type: New Feature
> Components: io-java-kafka, runner-flink, sdk-py-core
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
> Time Spent: 7h
> Remaining Estimate: 0h
>
> As of BEAM-6730, we can externally configure existing transforms from SDKs.
> We should add more useful transforms then just {{GenerateSequence}}.
> {{KafkaIO}} is a good candidate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
