Yes, it’s a known limitation [1] mostly due to the fact that KafkaIO.Read is based on UnboundedSource API and it fetches all information about topic and partitions only once during a “split" phase [2]. There is on-going work to make KafkaIO.Read based on Splittable DoFn [3] which should allow to get topic/partitions information dynamically, without restarting a pipeline.
[1] https://issues.apache.org/jira/browse/BEAM-727 [2] https://github.com/apache/beam/blob/8a54c17235f768f089b36265d79e69ee9b27a2ce/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedSource.java#L54 [3] https://issues.apache.org/jira/browse/BEAM-9977 <https://issues.apache.org/jira/browse/BEAM-9977> > On 29 Jun 2020, at 18:14, Talat Uyarer <tuya...@paloaltonetworks.com> wrote: > > Hi, > > I am using Dataflow. When I update my DF job with a new topic or update > partition count on the same topic. I can not use DF's update function. Could > you help me to understand why I can not use the update function for that ? > > I checked the Beam source code but I could not find the right place to read > on the code. > > Thanks for your help in advance > Talat
