Muthu, a) Not as of now. Please feel free to create the JIRA and specify the details there
b) I doubt increasing partitions will help. 500 GB/day/topic suggests the data per partition is only 10 GB/day. Before thinking about increasing the # of partitions, I would try a few things- 1. Inspect the consumer throughput metrics through the mbeans exposed on the Kafka consumers. 2. If individual consumer throughput looks reasonable, then deploy more consumer instances and see if that helps. Since you have 40-50 partitions per topic, you can have at least those many consumer instances. 3. If not, then check if the consumers post-process the data consumed from these partitions. If this processing is slow, your consumption rate will reduce. Thanks, Neha On Tue, Nov 20, 2012 at 3:12 AM, Muthukumar <zenmu...@gmail.com> wrote: > Hi Jun, > > Thanks for the response. > > a) Is there any plan in the roadmap to address this re-partition or > partition balance with new partitions? Please let me know to have the > JIRA for this. > > b) Do we need to go for more partitions for the topic6 (46 to ??) to > reduce the new requests + backlog. > > -Muthu > > On Tue, Nov 20, 2012 at 11:09 AM, Jun Rao <jun...@gmail.com> wrote: >> The configs look reasonable. Currently, we don't repartition existing data. >> Only new messages will consider the newly added partitions. >> >> Thanks >> >> Jun >>
