After brief search, I found KAFKA-6469
FYI
Original message From: Andrey Falko
Date: 2/1/18 5:28 PM (GMT-08:00) To: users@kafka.apache.org Subject: Re:
Recommended max number of topics (and data separation)
Indeed David, I confirmed that I can't push my clusters to more
Indeed David, I confirmed that I can't push my clusters to more than
72k topics with default zookeeper settings. Once I get to that
quantity, leader election never happens for new topics. Additionally
if I kill one of the brokers, all the topics don't get leaders
re-elected and it is impossible to
I used:
-Djute.maxbuffer=50111000
and the gain I had is that I could increment number of topics from 70k to
100k :P
2018-01-30 23:25 GMT+01:00 Andrey Falko :
> On Tue, Jan 30, 2018 at 1:38 PM, David Espinosa wrote:
> > Hi Andrey,
> > My topics are replicated with a replicated factor equals to th
On Tue, Jan 30, 2018 at 1:38 PM, David Espinosa wrote:
> Hi Andrey,
> My topics are replicated with a replicated factor equals to the number of
> nodes, 3 in this test.
> Didn't know about the kip-227.
> The problems I see at 70k topics coming from ZK are related to any
> operation where ZK has to
Hi Andrey,
My topics are replicated with a replicated factor equals to the number of
nodes, 3 in this test.
Didn't know about the kip-227.
The problems I see at 70k topics coming from ZK are related to any
operation where ZK has to retrieve topics metadata. Just listing topics at
50K or 60k you wil
On Sun, Jan 28, 2018 at 8:45 AM, David Espinosa wrote:
> Hi Monty,
>
> I'm also planning to use a big amount of topics in Kafka, so recently I
> made a test within a 3 nodes kafka cluster where I created 100k topics with
> one partition. Sent 1M messages in total.
Are your topic partitions replic
Hi Monty,
I'm also planning to use a big amount of topics in Kafka, so recently I
made a test within a 3 nodes kafka cluster where I created 100k topics with
one partition. Sent 1M messages in total.
These are my conclusions:
- There is not any limitation on kafka regarding the number of topic
I'm designing a system and need some more clarity regarding Kafka's
recommended limits on the number of topics and/or partitions. At a high
level, our system would work like this:
- A user creates a job X (X is a UUID).
- The user uploads data for X to an input topic: X.in.
- Workers process the d