HI Ondrej, You can retrieve metadata info using SDF (here is the video <https://www.youtube.com/watch?v=VQdtaaWxN0Y> from Israel Herraiz) and the kafka-python <https://kafka-python.readthedocs.io/en/master/> PY package.
Utkarsh On Tue, Mar 26, 2024 at 8:00 AM Ondřej Pánek <[email protected]> wrote: > Hello team, > > > > Is it possible to somehow retrieve metadata like topic, partition and > offset information from the consumed messages from Kafka source? I mean, if > that’s possible to do so in Python. I understand, that in Java, there is > the KafkaRecord construct, which offers these metadata, but I haven’t found > any alternative in Python. > > > > The reasoning is that we need to consume CDC data, where the offset > indicates the message’s order, i.e. the order of the CDC operation, and we > need to store it to process it later in BigQuery. > > > > Best regards, > > > > Ondrej >
