yeah..
I tried it with 10 messages with single broker and only one partiton that looked
instantaneous and ~5K messages/sec for the data size of 1KBI tried it with 1000
messages that looked instantaneous as well ~5K messages/sec for the data size of
1KBI tried it with 10K messages with single broker and only one partiton  things
started to go down ~1K messages/sec for the data size of 1KB
having only one partition on a single broker is a bad?  My goal is to run some
basic benchmarks on NATS & NSQ & KAFKA
I have the same environment for all three (NATS & NSQ & KAFKA)
a broker  on Machine 1producer on Machine 2Consumer on Machine 3
with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws instance.
I have pushed 300K messages with NATS and it was able to handle easily and
receive throughput was 5K messages/secI have pushed 300K messages and NSQ and
receive throughput was 2K messages/secI am unable to push 300K messages with
Kafka with the above configuration and environment so at this point my biggest
question is what is the fair setup for Kafka so its comparable with NATS and
NSQ?
kant
 





On Thu, Sep 15, 2016 12:43 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Lower the workload gradually, start from 10 messages, increase to 100, then

1000, and so on. See if it slows down as the workload increases. If so, you

need more brokers + partitions to handle the workload.




On Thu, Sep 15, 2016 at 12:42 PM, kant kodali <kanth...@gmail.com> wrote:




> m4.xlarge

>

>

>

>

>

>

> On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com

>

> wrote:

> What's the instance size that you're using? With 300k messages your single

>

> broker might not be able to handle it.

>

>

>

>

> On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com> wrote:

>

>

>

>

> My goal is to test the throughput (#messages per second) given my setup and

>>

>

> with a data size of 1KB. if you guys already have some idea on these

>>

>

> numbers

>>

>

> that would be helpful as well.

>>

>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

> On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com

>>

>

> wrote:

>>

>

> 172.* is all private ip's for my machine I double checked it.I have not

>>

>

> changed

>>

>

> any default settingsI dont know how to use kafka-consumer.sh

>>

>

> or kafka-producer.sh because it looks like they want me to specify a group

>>

>

> and I

>>

>

> didn't create any consumer group because I am using single producer and

>>

>

> consumer. is there a default group?Also, I am receiving message but very

>>

>

> late. I

>>

>

> send about 300K messages using the node.js client and I am receiving at a

>>

>

> very

>>

>

> low rate. really not sure what is going on?

>>

>

>

>>

>

>>

>

>>

>

>>

>

>>

>

>>

> On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com

>>

>

> wrote:

>>

>

> Your code seems to be using the public ip of the servers. If all 3 machines

>>

>

>

>>

> are in the same availability zone on AWS, try using the private ip, and

>>

>

>

>>

> then they might communicate over the local network.

>>

>

>

>>

>

>>

>

>>

>

>>

> Did you change any default settings?

>>

>

>

>>

>

>>

>

>>

>

>>

> Do you get the same results if you run kafka-consumer.sh and

>>

>

>

>>

> kafka-producer.sh instead of the Node code?

>>

>

>

>>

>

>>

>

>>

>

>>

> On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com> wrote:

>>

>

>

>>

>

>>

>

>>

>

>>

> > They are hosted on AWS and I dont think there are any network issues

>>

>

>

>>

> > because I

>>

>

>

>>

> > tried testing other Queuing systems with no issues however I am using a

>>

>

>

>>

> > node.js

>>

>

>

>>

> > client with the following code. I am not sure if there are any errors or

>>

>

>

>>

> > anything I didn't set in the following code?

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> > //producer var kafka = require('kafka-node'); var

>>

>

>

>>

> > Producer = kafka.Producer; var Client = kafka.Client; var client =

>>

>

>

>>

> > new Client('172.31.21.175:2181'); var argv =

>>

>

>

>>

> > require('optimist').argv; var topic = argv.topic || 'kafka_test'; var

>>

>

>

>>

> > p = argv.p || 0; var a = argv.a || 0; var producer = new

>>

>

>

>>

> > Producer(client, { requireAcks: 1}); var num = 300005;

>>

>

>

>>

> > producer.on('ready', function () { var message = 'Hello World';

>>

>

>

>>

> > for (var i=0; i<num; i++) { producer.send([ { topic:

>>

>

>

>>

> > topic, partition: p, messages: message, attributes: a } ], function

>>

>

>

>>

> > (err, result) { console.log(err || result);

>>

>

>

>>

> > //process.exit(); }); } }); producer.on('error',

>>

>

>

>>

> > function (err) { console.log('error', err); process.exit();

>>

>

>

>>

> > }); //Consumer var kafka = require('kafka-node'); var Consumer =

>>

>

>

>>

> > kafka.Consumer; var Offset = kafka.Offset; var Client =

>>

>

>

>>

> > kafka.Client; var argv = require('optimist').argv; var topic =

>>

>

>

>>

> > argv.topic || 'kafka_test'; var client = new

>>

>

>

>>

> > Client('172.31.21.175:2181'); var topics = [ {topic: topic,

>>

>

>

>>

> > partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:

>>

>

>

>>

> > 1000 }; var consumer = new Consumer(client, topics, options); var

>>

>

>

>>

> > offset = new Offset(client); var start; var received = 0; var

>>

>

>

>>

> > target = 200000; var hash = 1000; consumer.on('message', function

>>

>

>

>>

> > (message) { console.log(message); received += 1; if

>>

>

>

>>

> > (received === 1) { start = new Date(); } if (received === target) {

>>

>

>

>>

> > var stop = new Date(); console.log('\nDone test');

>>

>

>

>>

> > var mps = parseInt(target/((stop-start)/1000));

>>

>

>

>>

> > console.log('Received at ' + mps + ' msgs/sec'); process.exit();

>>

>

>

>>

> > } else if (received % hash === 0){

>>

>

>

>>

> > process.stdout.write(received + '\n'); } });

>>

>

>

>>

> > consumer.on('error', function (err) { console.log('error', err); });

>>

>

>

>>

> >

>>

>

>

>>

> > Not using Mixmax yet?

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> > On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com

>>

>

>

>>

> > wrote:

>>

>

>

>>

> > It sounds like a network issue. Where are the 3 servers located / hosted?

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> > On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth...@gmail.com>

>>

>

> wrote:

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

>>

> > Hi,

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > I have the following setup.

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > Single Kafka broker and Zookeeper on Machine 1single Kafka producer on

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > Machine 2

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > Single Kafka Consumer on Machine 3

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > When a producer client sends a message to the Kafka broker by pointing at

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > the

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > Zookeeper Server the consumer doesn't seem to get the message right away

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > instead

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > it gets after a minute or something (pretty late). I am not sure what

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > settings I

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > need to change. any ideas?

>>

>

>

>>

> >>

>>

>

>

>>

> >

>>

>

>

>>

> > Thanks,kant

>>

>

>

>>

> >

>>

>

>

>>

> >

>>

>

>

Reply via email to