Re: TopicFilters and 0.9 Consumer
Hi Guozhang, Thanks for confirming. It should be straightforward to make subscribe(TopicFilter) and subscribe(TopicFilter, Partition) work for added/removed topics, since this is mostly regex matching against zookeeper metadata. But any thoughts on how repartitioning would work? (we need to let the consumers know there are more partitions now, so they can spin up more consumers for these partitions). Is there a writeup of sorts somewhere, hinting at what to expect? Thanks Vinoth On Thu, Mar 5, 2015 at 8:56 AM, Guozhang Wang wangg...@gmail.com wrote: Vinoth, Yes we do have plans to continue supporting topic filters in 0.9 consumers, the APIs are not there yet though. Guozhang On Thu, Mar 5, 2015 at 8:32 AM, Vinoth Chandar vin...@uber.com wrote: Hi guys, I was wondering what the plan in 0.9, was for the topic filters that are today in the High level consumer. The new API' http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.html s subscribe methods, seem to be working with real topic names (which I personally like). Given, the topic filters today support automatic topic discovery (e.g.: MirrorMaker), would there be another mechanism to notify consumers of new topics and change in partitions? (I was thinking about an internal topic that provided these system changes, that consumers can listen to and choose whether or not they want to subscribe to say a newly created topic). Thanks Vinoth -- -- Guozhang
Re: TopicFilters and 0.9 Consumer
1. partition / member changes are caught on the server side, who will notify consumers to re-balance. 2. topic changes are caught on the client side through metadata request, who will then re-join the group with the new topic list to the server to re-balance. Guozhang On Fri, Mar 6, 2015 at 8:49 AM, Vinoth Chandar vin...@uber.com wrote: Hi Guozhang, Thanks for confirming. It should be straightforward to make subscribe(TopicFilter) and subscribe(TopicFilter, Partition) work for added/removed topics, since this is mostly regex matching against zookeeper metadata. But any thoughts on how repartitioning would work? (we need to let the consumers know there are more partitions now, so they can spin up more consumers for these partitions). Is there a writeup of sorts somewhere, hinting at what to expect? Thanks Vinoth On Thu, Mar 5, 2015 at 8:56 AM, Guozhang Wang wangg...@gmail.com wrote: Vinoth, Yes we do have plans to continue supporting topic filters in 0.9 consumers, the APIs are not there yet though. Guozhang On Thu, Mar 5, 2015 at 8:32 AM, Vinoth Chandar vin...@uber.com wrote: Hi guys, I was wondering what the plan in 0.9, was for the topic filters that are today in the High level consumer. The new API' http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.html s subscribe methods, seem to be working with real topic names (which I personally like). Given, the topic filters today support automatic topic discovery (e.g.: MirrorMaker), would there be another mechanism to notify consumers of new topics and change in partitions? (I was thinking about an internal topic that provided these system changes, that consumers can listen to and choose whether or not they want to subscribe to say a newly created topic). Thanks Vinoth -- -- Guozhang -- -- Guozhang
Re: TopicFilters and 0.9 Consumer
Vinoth, Yes we do have plans to continue supporting topic filters in 0.9 consumers, the APIs are not there yet though. Guozhang On Thu, Mar 5, 2015 at 8:32 AM, Vinoth Chandar vin...@uber.com wrote: Hi guys, I was wondering what the plan in 0.9, was for the topic filters that are today in the High level consumer. The new API' http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.html s subscribe methods, seem to be working with real topic names (which I personally like). Given, the topic filters today support automatic topic discovery (e.g.: MirrorMaker), would there be another mechanism to notify consumers of new topics and change in partitions? (I was thinking about an internal topic that provided these system changes, that consumers can listen to and choose whether or not they want to subscribe to say a newly created topic). Thanks Vinoth -- -- Guozhang
TopicFilters and 0.9 Consumer
Hi guys, I was wondering what the plan in 0.9, was for the topic filters that are today in the High level consumer. The new API' http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.htmls subscribe methods, seem to be working with real topic names (which I personally like). Given, the topic filters today support automatic topic discovery (e.g.: MirrorMaker), would there be another mechanism to notify consumers of new topics and change in partitions? (I was thinking about an internal topic that provided these system changes, that consumers can listen to and choose whether or not they want to subscribe to say a newly created topic). Thanks Vinoth
