KafkaIO has a readAll method that returns a PTransform<PCollection<KafkaSourceDescription>, PCollection<V>> is that what you mean? Then it could read in a 'dynamic' number of topics generated from somewhere else. Is that what you mean?
*~Vincent* On Tue, Dec 8, 2020 at 5:15 PM Daniel Collins <[email protected]> wrote: > /s/Combine/Flatten > > On Tue, Dec 8, 2020 at 8:06 PM Daniel Collins <[email protected]> > wrote: > >> Hi all, >> >> I'm trying to figure out if there's any possible use for reading from a >> dynamic set of Pub/Sub [Lite] subscriptions in a beam pipeline, although >> the same logic would apply to kafka topics. Does anyone know of a use case >> where you'd want to apply the same set of processing logic to all messages >> on a set of topics, but, you wouldn't know that set of topics when the >> pipeline is started? (otherwise you could just use Combine). >> >> -Dan >> >
