Hi Kim, You are correct in that the number of partitions sets the upper limit on consumer parallelization. That is, a single consumer in a group can consume multiple partitions, however multiple consumers in a group can't consume a single partition.
Also, since partitions are spread across your brokers, really it's the ratio nPartitions/nBrokers that you want to optimize for. Given the above parallelization limit, it would make sense to have a very large ratio. This would have other implications: - Your brokers will have a lot of smaller files to they will have to flush periodically. This can incur a lot of overhead and introduce latencies. Especially on a spinning disk where seeks are expensive. - Brokers are generally set to rotate their logs at a certain size. It could be hard to tune rotation with many small files. Given this, really you need to benchmark for your use case with your message sizes etc. Side-note: Note that for autoscaling you will have to overprovision your partitions somewhat to not hit the parallelization limit. Cheers, Jens On Wed, Mar 2, 2016 at 1:11 AM, BYEONG-GI KIM <bg...@bluedigm.com> wrote: > Hello. > > I have questions about how many partitions are optimal while using kafka. > As far as I know, even if there are multiple consumers that belong to a > consumer group, say *group_A*, only one consumer can receive a kafka > message produced by a producer if there is a partition. So, as a result, > multiple partitions are required in order to distribute the message to all > the consumers in group_A if I want the consumers to get the message. > > Is it right? > > I'm considering developing several kafka consumer applications, e.g., > message saver, message analyzer, etc., so a message from a producer must be > consumed by those kinds of consumers. > > Any advice and help would be really appreciated. > > Thanks in advance! > > Best regards > > Kim > -- Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook <https://www.facebook.com/#!/tink.se> Linkedin <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary> Twitter <https://twitter.com/tink>
