robertwb commented on issue #28664: URL: https://github.com/apache/beam/issues/28664#issuecomment-1743404283
Supporting the "raw" types as bytes certainly is the most flexible and makes sense as a first pass. I do think we'll want to support json and avro (at least) similar to the `_create_parser` idea we have for PubSub (I extended that for JSON at https://github.com/apache/beam/pull/28754 ), though for KafkaIO this'd be in Java of course. I just noticed that we already have https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java which is actually pretty complete. We'd probably want to add an option for the "raw" format that would pass the bytes through directly, as well as an option to add the key in if desired (similar to how the attributes can be appended as extra fields for PubSub). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
