pof-declaneaston commented on issue #25502:
URL: https://github.com/apache/beam/issues/25502#issuecomment-2438774490
Hi John. I have long moved on from this project so I can't tell you much. We
were using the built in Kafka consumer from the SDK, nothing custom.
This is how we setup the consumer.
`consumer_config = {
"bootstrap.servers": bootstrap_servers,
"auto.offset.reset": args.auto_offset_reset,
"group.id": args.groupid if args.groupid else
f"Application-{args.source_topic}",
}
producer_config = {
"bootstrap.servers": bootstrap_servers
}`
`ReadFromKafka(
consumer_config=consumer_config,
topics=[args.source_topic],
with_metadata=True,
max_num_records=args.max_kafka_records,
timestamp_policy=ReadFromKafka.create_time_policy)`
max_num_records is never set, that is for a "test mode".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
