Currently only some Beam sources are able to consume a configuration (set of topics here) that is dynamically generated and I don't think PubSubIO is one of them. So probably you'll have to implement a custom DoFn that reads from Cloud Pub/Sub to support this. Also, probably you'll have to constrain reading with size/time since Cloud Pub/Sub is an unbounded input source.
Thanks, Cham On Tue, Oct 31, 2023 at 2:22 PM Pravin DSouza <pravin...@gmail.com> wrote: > Hi, > > I have a use case where I want to listen to multiple Pub/Sub input topics > and route messages from all multiple input topics to a single destination > output topic. > The number of input topics can change any time and are stored in a table. > For example when I deploy for the first time, I might be reading from 3 > topics, but after 2 days the table is updated and now I want to read from 4 > topics.I don't want to redeploy the job because of a change in input topics > list. > I know how to use SideInputs for refresh, but I am unable to use SideInputs > to read the input topic details and pass to PubSubIO. > Can you please suggest which of the following ways this can be achieved if > new input topics are added: > 1. Using SideInput > 2. Any other approach > 3. Redeploy the job (this is the last course I want to rely on) > > Please suggest. > > Thank you, > Pravin >
