peay created BEAM-2449:
--------------------------
Summary: Support ValueProviders in KafkaIO
Key: BEAM-2449
URL: https://issues.apache.org/jira/browse/BEAM-2449
Project: Beam
Issue Type: Improvement
Components: runner-dataflow, sdk-java-extensions
Affects Versions: 2.0.0
Reporter: peay
Assignee: Thomas Groh
Priority: Minor
KafkaIO takes parameters at construction time:
- When partitions are provided, KafkaIO actually does nothing particular at
construction time.
- When only a topic is provided, I believe KafkaIO does fetch the list of
partitions at construction time.
Using ValueProvider for the brokers would be a good improvement. In particular,
when partitions are provided, a RuntimeProvider could be passed in.
Being able to always use a RuntimeProvider would be even better, such as when
using {{TextIO.xxx.withoutValidation()}}: this would considerably improve the
workflow of using Dataflow templates with KafkaIO, which currently requires to
hardcode brokers into the template.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
