[
https://issues.apache.org/jira/browse/BEAM-12076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361089#comment-17361089
]
Alexis De Man commented on BEAM-12076:
--------------------------------------
Having access to the messages metadata would allow us to make sure we are not
missing events in our destinations.
We consume Kafka data and stream it to BigQuery and it is easy to check that no
messages are missing using topic/partition/offset.
This is preventing us to move our consumers to Cloud Dataflow.
> Update Python cross-language Kafka source to read metadata
> ----------------------------------------------------------
>
> Key: BEAM-12076
> URL: https://issues.apache.org/jira/browse/BEAM-12076
> Project: Beam
> Issue Type: New Feature
> Components: cross-language, io-py-kafka
> Reporter: Chamikara Madhusanka Jayalath
> Priority: P2
> Labels: stale-P2
>
> Currently Python cross-language Kafka source [1] uses Java Kafka
> TypedWithoutMetadata transform [2]. We should update this so that users who
> need to use Kafka metadata in Python side have access to it.
> [1]
> https://github.com/apache/beam/blob/fd3075fa52303a0d684c2d7a91814eea5305476e/sdks/python/apache_beam/io/kafka.py#L107
> [2]
> https://github.com/apache/beam/blob/fd3075fa52303a0d684c2d7a91814eea5305476e/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L717
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
