m4.xlarge





On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
What's the instance size that you're using? With 300k messages your single

broker might not be able to handle it.




On Thu, Sep 15, 2016 at 12:30 PM, kant kodali <kanth...@gmail.com> wrote:




My goal is to test the throughput (#messages per second) given my setup and

with a data size of 1KB. if you guys already have some idea on these

numbers

that would be helpful as well.













On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com

wrote:

172.* is all private ip's for my machine I double checked it.I have not

changed

any default settingsI dont know how to use kafka-consumer.sh

or kafka-producer.sh because it looks like they want me to specify a group

and I

didn't create any consumer group because I am using single producer and

consumer. is there a default group?Also, I am receiving message but very

late. I

send about 300K messages using the node.js client and I am receiving at a

very

low rate. really not sure what is going on?













On Thu, Sep 15, 2016 12:06 AM, Ali Akhtar ali.rac...@gmail.com

wrote:

Your code seems to be using the public ip of the servers. If all 3 machines



are in the same availability zone on AWS, try using the private ip, and



then they might communicate over the local network.









Did you change any default settings?









Do you get the same results if you run kafka-consumer.sh and



kafka-producer.sh instead of the Node code?









On Thu, Sep 15, 2016 at 12:01 PM, kant kodali <kanth...@gmail.com> wrote:









> They are hosted on AWS and I dont think there are any network issues



> because I



> tried testing other Queuing systems with no issues however I am using a



> node.js



> client with the following code. I am not sure if there are any errors or



> anything I didn't set in the following code?



>



>



> //producer var kafka = require('kafka-node'); var



> Producer = kafka.Producer; var Client = kafka.Client; var client =



> new Client('172.31.21.175:2181'); var argv =



> require('optimist').argv; var topic = argv.topic || 'kafka_test'; var



> p = argv.p || 0; var a = argv.a || 0; var producer = new



> Producer(client, { requireAcks: 1}); var num = 300005;



> producer.on('ready', function () { var message = 'Hello World';



> for (var i=0; i<num; i++) { producer.send([ { topic:



> topic, partition: p, messages: message, attributes: a } ], function



> (err, result) { console.log(err || result);



> //process.exit(); }); } }); producer.on('error',



> function (err) { console.log('error', err); process.exit();



> }); //Consumer var kafka = require('kafka-node'); var Consumer =



> kafka.Consumer; var Offset = kafka.Offset; var Client =



> kafka.Client; var argv = require('optimist').argv; var topic =



> argv.topic || 'kafka_test'; var client = new



> Client('172.31.21.175:2181'); var topics = [ {topic: topic,



> partition: 0} ]; var options = { autoCommit: false, fetchMaxWaitMs:



> 1000 }; var consumer = new Consumer(client, topics, options); var



> offset = new Offset(client); var start; var received = 0; var



> target = 200000; var hash = 1000; consumer.on('message', function



> (message) { console.log(message); received += 1; if



> (received === 1) { start = new Date(); } if (received === target) {



> var stop = new Date(); console.log('\nDone test');



> var mps = parseInt(target/((stop-start)/1000));



> console.log('Received at ' + mps + ' msgs/sec'); process.exit();



> } else if (received % hash === 0){



> process.stdout.write(received + '\n'); } });



> consumer.on('error', function (err) { console.log('error', err); });



>



> Not using Mixmax yet?



>



>



>



>



>



>



>



> On Wed, Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com



> wrote:



> It sounds like a network issue. Where are the 3 servers located / hosted?



>



>



>



>



> On Thu, Sep 15, 2016 at 11:51 AM, kant kodali <kanth...@gmail.com>

wrote:



>



>



>



>



> Hi,



>>



>



> I have the following setup.



>>



>



> Single Kafka broker and Zookeeper on Machine 1single Kafka producer on



>>



>



> Machine 2



>>



>



> Single Kafka Consumer on Machine 3



>>



>



> When a producer client sends a message to the Kafka broker by pointing at



>>



>



> the



>>



>



> Zookeeper Server the consumer doesn't seem to get the message right away



>>



>



> instead



>>



>



> it gets after a minute or something (pretty late). I am not sure what



>>



>



> settings I



>>



>



> need to change. any ideas?



>>



>



> Thanks,kant



>



>

Reply via email to