Hi Vincent, ValueProviders required for Dataflow Classic templates.
Dataflow Flex templates supports I/O interfaces beyond ValueProvider, and here are some Kafka examples: * [1] Kafka to BigQuery Dataflow Flex template * [2] Kafka to Pub/Sub Dataflow Flex template * [3] Apache Beam pipeline example to ingest data from Apache Kafka to Google Cloud Pub/Sub * [4] Dataflow Classic and Flex templates info [1] https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/master/v2/kafka-to-bigquery [2] https://github.com/GoogleCloudPlatform/DataflowTemplates/tree/master/v2/kafka-to-pubsub [3] https://github.com/apache/beam/tree/master/examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub [4] https://cloud.google.com/dataflow/docs/concepts/dataflow-templates Hope this is helpful! Thanks, Alex From: Vincent Marquez <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, March 23, 2021 at 10:18 AM To: "[email protected]" <[email protected]> Subject: Make KafkaIO usable from Dataflow Template? Hello. I was looking at how to use the KafkaIO from a Dataflow Template, which requires all configuration options to be ValueProviders, which KafkaIO doesn't support currently. I saw this old PR: https://github.com/apache/beam/pull/6636/files I believe the reason this was never merged was there wasn't a good way to determine the number of splits to fire up in the UnboundedSource for KafkaIO. However, now that KafkaIO is based on SplittableDoFn which handles splitting in a dynamic fashion, are there still any blockers to this? Could we change all the KafkaIO parameters to ValueProviders now? ~Vincent
