Re: Round Robin Among Consumers

2015-07-25 Thread Gwen Shapira
I'm not sure the lessons here are completely generalizable. In this case there is a production system that works pretty well and that has many dependencies (i.e. many producers and consumers, some critical). I just needed to add a non-critical weekly (or possibly just one-time, if the POC does not

Re: Round Robin Among Consumers

2015-07-23 Thread Ewen Cheslack-Postava
Gwen, I'm curious about this use case. Given the Kafka -> HDFS flow, it obviously relates to Copycat. More generally, this could be a problem even when streaming data if your processing takes too long such that your consumer simply can't keep up with the rate at which messages are produced. The "

Re: Round Robin Among Consumers

2015-07-23 Thread Gwen Shapira
Agree. On Thu, Jul 23, 2015 at 2:43 PM, Jiangjie Qin wrote: > Ah, I see. Thanks for the use case, Gwen. I guess in that case it seems the > time to use low level consumer. > > Jiangjie (Becket) Qin > > On Thu, Jul 23, 2015 at 9:52 AM, Gwen Shapira wrote: > >> As crazy as it sounds, there is an a

Re: Round Robin Among Consumers

2015-07-23 Thread Jiangjie Qin
Ah, I see. Thanks for the use case, Gwen. I guess in that case it seems the time to use low level consumer. Jiangjie (Becket) Qin On Thu, Jul 23, 2015 at 9:52 AM, Gwen Shapira wrote: > As crazy as it sounds, there is an actual use-case there. > > Writing to HDFS can be very slow, so if you do a

Re: Round Robin Among Consumers

2015-07-23 Thread Gwen Shapira
As crazy as it sounds, there is an actual use-case there. Writing to HDFS can be very slow, so if you do a batch dump from a topic to HDFS, you may want more consumers reading from the topic than for "normal" streaming use-cases. We ended up modifying Camus to split a partition between multiple ma

Re: Round Robin Among Consumers

2015-07-23 Thread Jiangjie Qin
J A, It looks to me that in your case you actually want to scale the topic, right? Otherwise wouldn't a single consumer be enough? Jiangjie (Becket) Qin On Wed, Jul 22, 2015 at 7:39 PM, J A wrote: > Why have partition at all, if I don't need to scale topic. Coupling topic > scalability with co

Re: Round Robin Among Consumers

2015-07-22 Thread J A
Why have partition at all, if I don't need to scale topic. Coupling topic scalability with consumer scalability just goes against messaging systems core principle of decoupling consumer and producers On Wednesday, July 22, 2015, Aditya Auradkar wrote: > Hi, > > Why not simply have as many partit

Re: Round Robin Among Consumers

2015-07-22 Thread J A
Ordering gurantee should be optional. If someone needs ordering, they can always fall back on exclusive consumer strategy On Wednesday, July 22, 2015, Ashish Singh wrote: > Hey, don't you think that would be against the basic ordering guarantees > Kafka provides? > > On Wed, Jul 22, 2015 at 2:14

Re: Round Robin Among Consumers

2015-07-22 Thread Aditya Auradkar
Hi, Why not simply have as many partitions as the set of consumers you want to round robin across? Aditya On Wed, Jul 22, 2015 at 2:37 PM, Ashish Singh wrote: > Hey, don't you think that would be against the basic ordering guarantees > Kafka provides? > > On Wed, Jul 22, 2015 at 2:14 PM, J A

Re: Round Robin Among Consumers

2015-07-22 Thread Ashish Singh
Hey, don't you think that would be against the basic ordering guarantees Kafka provides? On Wed, Jul 22, 2015 at 2:14 PM, J A wrote: > Hi, This is reference to stackoverflow question " > > http://stackoverflow.com/questions/31547216/kafka-log-deletion-and-load-balancing-across-consumers > " > Si

Round Robin Among Consumers

2015-07-22 Thread J A
Hi, This is reference to stackoverflow question " http://stackoverflow.com/questions/31547216/kafka-log-deletion-and-load-balancing-across-consumers " Since Kafka 0.8 already maintains a client offset, i would like to request a feature, where a single partition consumption can be round robin across