Re: Apache Kafka blog on more partitions support

2018-11-06 Thread Jun Rao
The blog is now published to https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions Thanks, Jun On Fri, Nov 2, 2018 at 2:36 PM, Jun Rao wrote: > Hi, Mayuresh, > > Most of the controlled shutdown time is in leader election. The controller > currently doesn't wait for

Re: Apache Kafka blog on more partitions support

2018-11-02 Thread Jun Rao
Hi, Mayuresh, Most of the controlled shutdown time is in leader election. The controller currently doesn't wait for LeaderAndIsrRequest to be sent out before responding to the controlled shutdown request. Thanks, Jun On Fri, Nov 2, 2018 at 1:58 PM, Mayuresh Gharat wrote: > Thanks Jun for

Re: Apache Kafka blog on more partitions support

2018-11-02 Thread Jun Rao
Hi, Harsha, The recommendation is the upper bound for partitions. One can certainly have more than 50 brokers in a cluster if there are fewer partitions per broker. I will clarify this in the blog. Thanks, Jun On Fri, Nov 2, 2018 at 12:07 PM, wrote: > Thanks Jun for sharing the post. > Minor

Re: Apache Kafka blog on more partitions support

2018-11-02 Thread Mayuresh Gharat
Thanks Jun for sharing this. Looks nice ! Do we intend to shed light on how much time is required, on an average, for new Leader election. Also would it be good to add "if the controller waits for the LeaderAndIsrResponses before sending shutDown_OK to the shutting down broker". Thanks,

Re: Apache Kafka blog on more partitions support

2018-11-02 Thread harshach
Thanks Jun for sharing the post. Minor Nit: Date says  December 16, 2019. Did this test measured the replication affects on the overall cluster health and performance? It looks like we are suggesting with 200k partitions and 4k per broker max size of a cluster should be around 50 brokers?