Thanks Cham. We analyzed and decided our requirement is not worth the effort needed to handle the unbounded Cloud PubSub input source. Since our input list will not change frequently, we are considering redeployment as better option for now.
Regards, Pravin On Wed, Nov 1, 2023 at 2:06 PM Chamikara Jayalath via user < user@beam.apache.org> wrote: > Currently only some Beam sources are able to consume a configuration (set > of topics here) that is dynamically generated and I don't think PubSubIO is > one of them. So probably you'll have to implement a custom DoFn that reads > from Cloud Pub/Sub to support this. Also, probably you'll have to constrain > reading with size/time since Cloud Pub/Sub is an unbounded input source. > > Thanks, > Cham > > > > On Tue, Oct 31, 2023 at 2:22 PM Pravin DSouza <pravin...@gmail.com> wrote: > >> Hi, >> >> I have a use case where I want to listen to multiple Pub/Sub input topics >> and route messages from all multiple input topics to a single destination >> output topic. >> The number of input topics can change any time and are stored in a table. >> For example when I deploy for the first time, I might be reading from 3 >> topics, but after 2 days the table is updated and now I want to read from >> 4 >> topics.I don't want to redeploy the job because of a change in input >> topics >> list. >> I know how to use SideInputs for refresh, but I am unable to use >> SideInputs >> to read the input topic details and pass to PubSubIO. >> Can you please suggest which of the following ways this can be achieved if >> new input topics are added: >> 1. Using SideInput >> 2. Any other approach >> 3. Redeploy the job (this is the last course I want to rely on) >> >> Please suggest. >> >> Thank you, >> Pravin >> >
