Re: why did Kafka choose pull instead of push for a consumer ?

Hmm...Looks like Kafka is written in Scala. There is this thing called reactive streams where a slow consumer can apply back pressure if they are consuming slow. Even with Java this is possible with a Library called RxJava and these ideas will be incorporated in Java 9 as well. I still don't see

Re: why did Kafka choose pull instead of push for a consumer ?

Still it should be possible to implement using reactive streams right. Could you please enlighten me on what are the some major differences you see between a commit log and a message queue? I see them being different only in the implementation but not functionality wise so I would be glad to hear

Re: any update on this?

+2 watching. On Sat, Sep 17, 2016 at 2:45 AM, kant kodali wrote: > https://issues.apache.org/jira/browse/KAFKA-1793 > It would be great to use Consul instead of Zookeeper for Kafka and I think > it > would benefit Kafka a lot from the exponentially growing consul community.

Re: why did Kafka choose pull instead of push for a consumer ?

Kafka is not a queue. It’s a distributed commit log. – Best regards, Radek Gruchalski ra...@gruchalski.com On September 17, 2016 at 9:23:09 PM, kant kodali (kanth...@gmail.com) wrote: Hmm...Looks like Kafka is written in Scala. There is this thing called reactive streams where a slow consumer

Re: why did Kafka choose pull instead of push for a consumer ?

There are two distinct questions... 1. Regarding reactive streams, Akka has an implementation for Kafka: https://github.com/akka/reactive-kafka 2. Kafka is not a queue. For example, it does not implement "dequeue" operation. All the message management / retention is not based on whether a message

Re: why did Kafka choose pull instead of push for a consumer ?

Please read this article: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying – Best regards, Radek Gruchalski ra...@gruchalski.com On September 17, 2016 at 9:49:43 PM, kant kodali (kanth...@gmail.com) wrote: Still it

Re: can one topic be registered in multiple brokers?

so Zookeeper will select which broker it should direct the message to if I have 3 brokers and 3 partitions of a topic? I only finished the benchmarks of Kafka and NSQ and still working NATS Streaming Server (one more day I will finish it). But so far Kafka had a great throughput with Java Client

why did Kafka choose pull instead of push for a consumer ?

why did Kafka choose pull instead of push for a consumer? push sounds like it is more realtime to me than poll and also wouldn't poll just keeps polling even when they are no messages in the broker causing more traffic? please enlighten me

any update on this?

https://issues.apache.org/jira/browse/KAFKA-1793 It would be great to use Consul instead of Zookeeper for Kafka and I think it would benefit Kafka a lot from the exponentially growing consul community.

can one topic be registered in multiple brokers?

can one topic be registered in multiple brokers? if so, which component of kafka decides which broker should get the message for that particular topic? Thanks!

Re: can one topic be registered in multiple brokers?

Kafka uses murmur2 key hashing by default. You can also create your own custom partitioner. The partitioner can be set on per producer / consumer basis. -- Best regards, Rad On Sat, Sep 17, 2016 at 11:01 AM +0200, "kant kodali" wrote: so Zookeeper will select

Re: why did Kafka choose pull instead of push for a consumer ?

I'm only guessing here regarding if this is the reason: Pull is much more sensible when a lot of data is pushed through. It allows consumers consuming at their own pace, slow consumers do not slow the complete system down. -- Best regards, Rad On Sat, Sep 17, 2016 at 11:18 AM +0200, "kant

Re: Kafka consumer group problem

Did you try props.put("group.id", "test"); On Thu, Sep 15, 2016 at 12:55 AM, Joyce Chen wrote: > Hi, > > I created a few consumers that belong to the same group_id, but I noticed > that each consumer get all messages instead of only some of the messages. > > As for the

Re: Performance issue with KafkaStreams

Hi Caleb, I usually do './gradlew installAll' first and that places all the jars in my local maven repo in ~/.m2/repository. Eno > On 17 Sep 2016, at 00:30, Caleb Welton wrote: > > Is there a specific way that I need to build kafka for that to work? > > bash$ export