Please read this article: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
– Best regards, Radek Gruchalski ra...@gruchalski.com On September 17, 2016 at 9:49:43 PM, kant kodali (kanth...@gmail.com) wrote: Still it should be possible to implement using reactive streams right. Could you please enlighten me on what are the some major differences you see between a commit log and a message queue? I see them being different only in the implementation but not functionality wise so I would be glad to hear your thoughts. On Sat, Sep 17, 2016 12:39 PM, Radoslaw Gruchalski ra...@gruchalski.com wrote: Kafka is not a queue. It’s a distributed commit log. – Best regards, Radek Gruchalski ra...@gruchalski.com On September 17, 2016 at 9:23:09 PM, kant kodali (kanth...@gmail.com) wrote: Hmm...Looks like Kafka is written in Scala. There is this thing called reactive streams where a slow consumer can apply back pressure if they are consuming slow. Even with Java this is possible with a Library called RxJava and these ideas will be incorporated in Java 9 as well. I still don't see why they would pick poll just to solve this one problem and compensating on others. Poll just don't sound realtime. I heard from some people that they would set poll to 100ms. Well 1) that is a lot of time. 2) Financial applications requires micro second latency. Kafka from what I understand looks like has a very high latency and here is the article. http://bravenewgeek.com/dissecting-message-queues/ I usually don't go by articles but I ran my own experiments on different queues and my numbers are very close to this article so I would say whoever wrote this article has done a good Job. 3) poll does generate unnecessary traffic in case if the data isn't available. Finally still not sure why they would pick poll() ? or do they plan on introducing reactive streams?Thanks,kant On Sat, Sep 17, 2016 5:14 AM, Radoslaw Gruchalski ra...@gruchalski.com wrote: I'm only guessing here regarding if this is the reason: Pull is much more sensible when a lot of data is pushed through. It allows consumers consuming at their own pace, slow consumers do not slow the complete system down. -- Best regards, Rad On Sat, Sep 17, 2016 at 11:18 AM +0200, "kant kodali" <kanth...@gmail.com> wrote: why did Kafka choose pull instead of push for a consumer? push sounds like it is more realtime to me than poll and also wouldn't poll just keeps polling even when they are no messages in the broker causing more traffic? please enlighten me
