RE: Partitions and consumer assignment

My mistake, I missed the carriage return -Original Message- From: Heath Ivie [mailto:hi...@autoanything.com] Sent: Tuesday, January 19, 2016 2:58 PM To: users@kafka.apache.org Subject: RE: Partitions and consumer assignment I get a 404 for this page. Is there an updated link

RE: Partitions and consumer assignment

I get a 404 for this page. Is there an updated link? -Original Message- From: Jason J. W. Williams [mailto:jasonjwwilli...@gmail.com] Sent: Tuesday, January 19, 2016 1:33 PM To: users@kafka.apache.org Subject: Re: Partitions and consumer assignment Thanks Luke. I'll read through

Re: Partitions and consumer assignment

Thanks Luke. I'll read through that. -J On Tue, Jan 19, 2016 at 9:21 AM, Luke Steensen < luke.steen...@braintreepayments.com> wrote: > Hi Jason, > > You might find this blog post useful: > > http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ > > - Luke

Re: Partitions and consumer assignment

Hi Jason, You might find this blog post useful: http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ - Luke On Sat, Jan 16, 2016 at 1:30 PM, Jason Williams wrote: > Hi Franco, > > Thank you for the info! It is my reading of the docs that adding > parti

Re: Partitions and consumer assignment

Hi Franco, Thank you for the info! It is my reading of the docs that adding partitions would require manual rebalancing to spread the existing load? Would it be advisable instead to start out with a partition count that is 10x the initial consumer count? For example if we anticipate starting wi

Re: Partitions and consumer assignment

Hi Jason, You can try to repartition and assign more consumers. *Documentation* "Modifying topics You can change the configuration or partitioning of a topic using the same topic tool. To add partitions you can do > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_n

Re: Partitions and consumer assignment

Hi Franco, Thanks for responding but that doesn't really answer my question. The situation described is it I have N partitions and already N consumers in the CG, and then I receive a spike in messages. How is it suggested to handle adding more consumption capacity to deal the spike...since add

Re: Partitions and consumer assignment

1 consumer group can have many partitions, if the consumer group has 1 consumer and there are N partitions, it will consume from N, if you have a spike you can add up to N more consumers to that consumer group. 2016-01-16 11:32 GMT+01:00 Jason Williams : > Thanks Jens! > > So assuming you've alre

Re: Partitions and consumer assignment

Thanks Jens! So assuming you've already paired your partition count to the consumer count...if you experience a spike in messages and want to spin up more consumers to add temporary processing capacity, what's the suggested way to handle this? (since it would seem you can't just add consumers a

Re: Partitions and consumer assignment

Hi, You are correct. The others will remain idle. This is why you generally want to have at least the same number of partitions as consumers. Cheers, Jens – Skickat från Mailbox On Sat, Jan 16, 2016 at 2:34 AM, Jason J. W. Williams wrote: > Hi, > I'm trying to make sure I understand