In Kafka you can use the WhiteList and BlackList topic filters (to regex topics that match or do not match respectively). The console consumer is a nice example how to-do this with the high level consumer https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsoleConsumer.scala#L137then when creating a stream that you can iterate over using that filterSpec https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsoleConsumer.scala#L196and you are good to go.
/******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Mon, May 12, 2014 at 3:10 AM, David Miller <[email protected]>wrote: > > The kafka "high level" consumer supports topic filtering for doing > wildcards against topics. > The storm-kafka spout uses the simple consumer which doesnt have these > features (but has others needed for reliable messaging, offset management) > > you could write your own high level consumer spout which could support > this feature at the expense of some message reliability. > > Look at TopicFilter > https://kafka.apache.org/08/api.html > > > > > > > On Mon, May 12, 2014 at 4:59 PM, Amikam Snir <[email protected]> wrote: > >> Hi all, >> >> Do you know code (Spout-code), which consume from multiple topics for >> example using wildcard against the topic name? >> >> >> >> The topics at my application are created dynamically. Should I just use >> dedicated Kafka-Spout for each topic and somehow updated the topology at >> run-time? >> >> >> >> >> >> Thanks in advance, >> >> Amikam. >> > >
