Re: can one topic be registered in multiple brokers?
Kafka uses murmur2 key hashing by default. You can also create your own custom partitioner. The partitioner can be set on per producer / consumer basis. -- Best regards, Rad On Sat, Sep 17, 2016 at 11:01 AM +0200, "kant kodali" wrote: so Zookeeper will select which broker it should direct the message to if I have 3 brokers and 3 partitions of a topic? I only finished the benchmarks of Kafka and NSQ and still working NATS Streaming Server (one more day I will finish it). But so far Kafka had a great throughput with Java Client about 55000 messages per second(receive throughput) and 70K messages per second(sendthrougput)each message size is 1KB. so yeah something wrong with node.js client NSQ was ~ 3K receive throughput and~4K send throughput one thing about my NSQ benchmark is first of all I am new to Go and I have the following code (pasting only the lines that are relevant). I just dont know if w.Publish is a blocking call? If it is a blocking call I have to see if the library offers any non-blocking async call since the java version I wrote for Kafka was async. That way I am comparing apples to apples w, _ := nsq.NewProducer("172.31.18.175:4150", config) for i := 0; i<50; i++ { err := w.Publish("nsq_go_test", []byte(data)) if err != nil { log.Panic("Could not connect") } } >From what I heard beating NATS is going to be harder but I rather run my own experiment than going with what I heard. Also it On Sat, Sep 17, 2016 1:38 AM, Ali Akhtar ali.rac...@gmail.com wrote: You can create multiple partitions of a topic and kafka will attempt to distribute them evenly. E.g if you have 3 brokers and you create 3 partitions of a topic, each broker will be the leader of 1 of the 3 partitions. P.S how did the benchmarking go? On Sat, Sep 17, 2016 at 1:36 PM, kant kodali wrote: > can one topic be registered in multiple brokers? if so, which component of > kafka decides which broker should get the message for that particular > topic? > Thanks!
Re: can one topic be registered in multiple brokers?
so Zookeeper will select which broker it should direct the message to if I have 3 brokers and 3 partitions of a topic? I only finished the benchmarks of Kafka and NSQ and still working NATS Streaming Server (one more day I will finish it). But so far Kafka had a great throughput with Java Client about 55000 messages per second(receive throughput) and 70K messages per second(sendthrougput)each message size is 1KB. so yeah something wrong with node.js client NSQ was ~ 3K receive throughput and~4K send throughput one thing about my NSQ benchmark is first of all I am new to Go and I have the following code (pasting only the lines that are relevant). I just dont know if w.Publish is a blocking call? If it is a blocking call I have to see if the library offers any non-blocking async call since the java version I wrote for Kafka was async. That way I am comparing apples to apples w, _ := nsq.NewProducer("172.31.18.175:4150", config) for i := 0; i<50; i++ { err := w.Publish("nsq_go_test", []byte(data)) if err != nil { log.Panic("Could not connect") } } >From what I heard beating NATS is going to be harder but I rather run my own experiment than going with what I heard. Also it On Sat, Sep 17, 2016 1:38 AM, Ali Akhtar ali.rac...@gmail.com wrote: You can create multiple partitions of a topic and kafka will attempt to distribute them evenly. E.g if you have 3 brokers and you create 3 partitions of a topic, each broker will be the leader of 1 of the 3 partitions. P.S how did the benchmarking go? On Sat, Sep 17, 2016 at 1:36 PM, kant kodali wrote: > can one topic be registered in multiple brokers? if so, which component of > kafka decides which broker should get the message for that particular > topic? > Thanks!
Re: can one topic be registered in multiple brokers?
You can create multiple partitions of a topic and kafka will attempt to distribute them evenly. E.g if you have 3 brokers and you create 3 partitions of a topic, each broker will be the leader of 1 of the 3 partitions. P.S how did the benchmarking go? On Sat, Sep 17, 2016 at 1:36 PM, kant kodali wrote: > can one topic be registered in multiple brokers? if so, which component of > kafka decides which broker should get the message for that particular > topic? > Thanks!
can one topic be registered in multiple brokers?
can one topic be registered in multiple brokers? if so, which component of kafka decides which broker should get the message for that particular topic? Thanks!