Hi, Another idea. Less cost effective, but easier to implement and maintain: 1. Read from all possible topics. 2. 'Map' to add constant topic name to every topic. (Or some other variations of this step) 3. 'Filter' with side input to just filter out all topics that should be silenced. 4. 'Flatten' But this requires that new topics are not created during run of the pipeline and You know them. If not then suggestion with custom 'ParDo' is only option, but I would also suggest rethinking Your infrastructure setup.
Best Wiśniowski Piotr śr., 1 lis 2023, 19:06 użytkownik Chamikara Jayalath via user < user@beam.apache.org> napisał: > Currently only some Beam sources are able to consume a configuration (set > of topics here) that is dynamically generated and I don't think PubSubIO is > one of them. So probably you'll have to implement a custom DoFn that reads > from Cloud Pub/Sub to support this. Also, probably you'll have to constrain > reading with size/time since Cloud Pub/Sub is an unbounded input source. > > Thanks, > Cham > > > > On Tue, Oct 31, 2023 at 2:22 PM Pravin DSouza <pravin...@gmail.com> wrote: > >> Hi, >> >> I have a use case where I want to listen to multiple Pub/Sub input topics >> and route messages from all multiple input topics to a single destination >> output topic. >> The number of input topics can change any time and are stored in a table. >> For example when I deploy for the first time, I might be reading from 3 >> topics, but after 2 days the table is updated and now I want to read from >> 4 >> topics.I don't want to redeploy the job because of a change in input >> topics >> list. >> I know how to use SideInputs for refresh, but I am unable to use >> SideInputs >> to read the input topic details and pass to PubSubIO. >> Can you please suggest which of the following ways this can be achieved if >> new input topics are added: >> 1. Using SideInput >> 2. Any other approach >> 3. Redeploy the job (this is the last course I want to rely on) >> >> Please suggest. >> >> Thank you, >> Pravin >> >
