Hello,
I’m encountering the following error while working with ReadFromKafka with the
Dataflow runner V2 in Apache Beam:
2025-02-25 23:14:19.435 PST
Error translating pipeline. Unknown non-composite transform urn
beam:transform:read:v1 for
external_1Reading-messages-from-Kafka-KafkaIO-Read-KafkaIO-Read-ReadFromKafkaViaUnbounded-Read-KafkaUnboundedS
Below is the code snippet triggering the error:
from apache_beam.io.kafka import ReadFromKafka, default_io_expansion_service
pcoll = self.pipeline | "Reading messages from Kafka" >>
(ReadFromKafka(
self.consumer_config,
self.kafka_config.get('topics'),
commit_offset_in_finalize=True,
with_metadata=True,
expansion_service=default_io_expansion_service(append_args=['--experiments=use_deprecated_read']))
)
Background:
I’m looking to switch from the default connector (ReadFromKafkaViaSDF) to the
legacy connector (ReadFromKafkaViaUnboundedSource), as the default connector is
causing the Dataflow job to generate many Kafka connections.
Any guidance or suggestions to resolve this issue would be greatly appreciated.
Thank you
--
Utkarsh
Sr. Data Engineer
Albertsons Companies
Mobile: (860)-578-5208
Email: [email protected]
________________________________
Warning: All e-mail sent to this address will be received by the corporate
e-mail system, and is subject to archival and review by someone other than the
recipient. This e-mail may contain proprietary information and is intended only
for the use of the intended recipient(s). If the reader of this message is not
the intended recipient(s), you are notified that you have received this message
in error and that any review, dissemination, distribution or copying of this
message is strictly prohibited. If you have received this message in error,
please notify the sender immediately.
________________________________
