[
https://issues.apache.org/jira/browse/BEAM-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657085#comment-16657085
]
Raghu Angadi commented on BEAM-5786:
------------------------------------
It will cause reader to fail. Streaming runners typically keep trying and the
reader will keep failing. Watermark will be stuck.
Moving to SDF makes it possible for KafkaIO to remove splits at runtime if
there are no partitions assigned to it, that way we don't worry about what
watermark to report for such splits.
Overall supporting watermark correctly for a topic that might appear in future
is impossible since we don't know anything about what the timestamp in it will
be. This is a generic problem for all sources that might find new input at
runtime, not just for KafkaIO.
> KafkaIO should support topic patterns
> -------------------------------------
>
> Key: BEAM-5786
> URL: https://issues.apache.org/jira/browse/BEAM-5786
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
> Reporter: Alexey Romanenko
> Assignee: Alexey Romanenko
> Priority: Major
>
> For the moment, it is possible to set topics for KafkaIO.Read in 2 ways - one
> single topic by using {{withTopic(String)}} or list of topics by using
> {{withTopics(List<String>)}}.
> It would make sense to provide another way for this using patterns, based on
> regular expressions, like {{withTopics(“org_[0-9]+_source”)}}. In this case,
> Kafka consumer could discover the topics based on user provided pattern.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
