Hi Jakob, currently, its not possible to subscribe to multiple topics with one FlinkKafkaConsumer.
So for now, you have to create a FKC for each topic .. so you'll end up with 50 sources. As soon as Kafka releases the new consumer, it will support subscribing to multiple topics (I think even with pattern support) and we can easily expose the APIs also to the FlinkKafkaConsumer. As you can see here: https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan Kafka has plans to release the new consumer API in October. As soon as the new API is out, we'll support it. I hope this solution is okay for you. If not, please let me know ;) Robert On Fri, Sep 18, 2015 at 10:43 AM, Jakob Ericsson <jakob.erics...@gmail.com> wrote: > Hi, > > Would it be possible to get the FlinkKafkaConsumer to support multiple > topics, like a list? > > Or would it be better to instantiate one FlinkKafkaConsumers per topic and > add as a source? > We have about 40-50 topics to listen for one job. > Or even better, supply a regexp pattern that defines the queues, this > means that you have to do some queries against ZK to get information about > topics. > > Kind regards, > Jakob >
