Re: Kafka New Consumer Performance Test ThroughPut Degradation

Hi Jason, Thanks for the update, I will be watching KAFKA-2489. Thanks, -Poorna On Wed, Sep 2, 2015 at 5:31 PM, Jason Gustafson wrote: > Hi Poorna, > > We've been looking at the same issue as part of KAFKA-2489. I suspect the > initial rebalance is causing the test to

Re: Kafka New Consumer Performance Test ThroughPut Degradation

Hi, I have checked out the latest code out of https://github.com/apache/kafka based on commit id e582447adb4708731aff74aa294e7ce2b30b0a41. Looks like the performance test on the new-consumer is broken. in/kafka-consumer-perf-test.sh --zookeeper zkip:2181 --broker-list brokerIp:9092 --topic test

Re: Kafka New Consumer Performance Test ThroughPut Degradation

Hi Poorna, We've been looking at the same issue as part of KAFKA-2489. I suspect the initial rebalance is causing the test to timeout, but we're still investigating. Thanks, Jason On Wed, Sep 2, 2015 at 2:03 PM, Poorna Chandra Tejashvi Reddy < pctre...@gmail.com> wrote: > Hi, > > I have

Re: Kafka New Consumer Performance Test ThroughPut Degradation

Tried bisecting, but turns out things were broken for some time. We really need some system tests in place to avoid letting even new code break for so long. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good performance - an error due to broker apparently not accepting the

Re: Kafka New Consumer Performance Test ThroughPut Degradation

So I’ve been looking at the SSL performance (which potentially also has an issue unrelated to this). I’d noticed strange behaviour with poll() on the new consumer but I wasn’t sure whether this was a bug or a feature. On closer inspection it seems to arise from ConsumerPerformance relying on a

Kafka New Consumer Performance Test ThroughPut Degradation

Hi, We have built the latest kafka from https://github.com/apache/kafka based on this commit id 436b7ddc386eb688ba0f12836710f5e4bcaa06c8 . We ran the performance test on a 3 node kafka cluster. There is a huge throughput degradation using the new-consumer compared to the regular consumer. Below

Re: Kafka New Consumer Performance Test ThroughPut Degradation

I don't think the commands are really equivalent despite just adding the --new-consumer flag. ConsumerPerformance uses a single thread when using the new consumer (it literally just allocates the consumer, loops until it's consumed enough, then exits), whereas the old consumer uses a bunch of

Re: Kafka New Consumer Performance Test ThroughPut Degradation

I think this is likely a regression. The two clients had more or less equivalent performance when we checked in the code (see my post on this earlier in the year). Looks like maybe we broke something up in the interim? On my laptop the new consumer perf seems to have dropped from about ~200MB/sec

Re: Kafka New Consumer Performance Test ThroughPut Degradation

436b7ddc386eb688ba0f12836710f5e4bcaa06c8 is pretty recent and there could be some current consumer improvement patches that introduces some regression. I would suggest doing a binary search in the log from 3f8480ccfb011eb43da774737597c597f703e11b (maybe even earlier?) to do a quick check.