Deleting topics in windows - fix estimate or workaround

2019-11-30 Thread Sachin Mittal
Hi All, I hope we are well aware of the critical bug in windows where kafka crashes when we delete a topic. This affects other areas too like in stream processing when trying to reset a stream using StreamsResetter. Right now only workaround I have found is to stop zookeeper and kafka server and

RE: More partitions => less throughput?

2019-11-30 Thread Eric Owhadi
What is happening imho is that when you have multiple partitions, each consumer will fetch data from its partition and find only 1/64th the amount of data (compared to the single partition case) to send every time it is its turn to send stuff. Therefore you end up having a more chatty

[VOTE] 2.4.0 RC2

2019-11-30 Thread Manikumar
Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 2.4.0. This release includes many new features, including: - Allow consumers to fetch from closest replica - Support for incremental cooperative rebalancing to the consumer rebalance

Re: More partitions => less throughput?

2019-11-30 Thread Peter Bukowinski
Testing multiple brokers VMs on a single host won’t give you accurate performance numbers unless that is how you will be deploying kafka in production. (Don’t do this.) All your kafka networking is being handled by a single host, so instead of being spread out between machines to increase total

Re: More partitions => less throughput?

2019-11-30 Thread Tom Brown
I think the number of partitions needs to be tuned to the size of the cluster; 64 partitions on what is essentially a single box seems high. Do you know what hardware you will be deploying on in production? Can you run your benchmark on that instead of a vm? —Tom On Thursday, November 28, 2019,

Re: [VOTE] 2.4.0 RC1

2019-11-30 Thread Manikumar
Hi All, We will consider KAFKA-9244 as blocker and include the fix in 2.4 release. I am canceling this VOTE and will create third release candidate. Thank you all for testing. On Fri, Nov 29, 2019 at 10:52 AM Matthias J. Sax wrote: > I did