I'm not sure the lessons here are completely generalizable.
In this case there is a production system that works pretty well and
that has many dependencies (i.e. many producers and consumers, some
critical). I just needed to add a non-critical weekly (or possibly
just one-time, if the POC does not
Gwen,
I'm curious about this use case. Given the Kafka -> HDFS flow, it obviously
relates to Copycat. More generally, this could be a problem even when
streaming data if your processing takes too long such that your consumer
simply can't keep up with the rate at which messages are produced.
The "
Agree.
On Thu, Jul 23, 2015 at 2:43 PM, Jiangjie Qin wrote:
> Ah, I see. Thanks for the use case, Gwen. I guess in that case it seems the
> time to use low level consumer.
>
> Jiangjie (Becket) Qin
>
> On Thu, Jul 23, 2015 at 9:52 AM, Gwen Shapira wrote:
>
>> As crazy as it sounds, there is an a
Ah, I see. Thanks for the use case, Gwen. I guess in that case it seems the
time to use low level consumer.
Jiangjie (Becket) Qin
On Thu, Jul 23, 2015 at 9:52 AM, Gwen Shapira wrote:
> As crazy as it sounds, there is an actual use-case there.
>
> Writing to HDFS can be very slow, so if you do a
As crazy as it sounds, there is an actual use-case there.
Writing to HDFS can be very slow, so if you do a batch dump from a
topic to HDFS, you may want more consumers reading from the topic than
for "normal" streaming use-cases. We ended up modifying Camus to split
a partition between multiple ma
J A,
It looks to me that in your case you actually want to scale the topic,
right? Otherwise wouldn't a single consumer be enough?
Jiangjie (Becket) Qin
On Wed, Jul 22, 2015 at 7:39 PM, J A wrote:
> Why have partition at all, if I don't need to scale topic. Coupling topic
> scalability with co
Why have partition at all, if I don't need to scale topic. Coupling topic
scalability with consumer scalability just goes against messaging systems
core principle of decoupling consumer and producers
On Wednesday, July 22, 2015, Aditya Auradkar
wrote:
> Hi,
>
> Why not simply have as many partit
Ordering gurantee should be optional. If someone needs ordering, they can
always fall back on exclusive consumer strategy
On Wednesday, July 22, 2015, Ashish Singh wrote:
> Hey, don't you think that would be against the basic ordering guarantees
> Kafka provides?
>
> On Wed, Jul 22, 2015 at 2:14
Hi,
Why not simply have as many partitions as the set of consumers you want to
round robin across?
Aditya
On Wed, Jul 22, 2015 at 2:37 PM, Ashish Singh wrote:
> Hey, don't you think that would be against the basic ordering guarantees
> Kafka provides?
>
> On Wed, Jul 22, 2015 at 2:14 PM, J A
Hey, don't you think that would be against the basic ordering guarantees
Kafka provides?
On Wed, Jul 22, 2015 at 2:14 PM, J A wrote:
> Hi, This is reference to stackoverflow question "
>
> http://stackoverflow.com/questions/31547216/kafka-log-deletion-and-load-balancing-across-consumers
> "
> Si
Hi, This is reference to stackoverflow question "
http://stackoverflow.com/questions/31547216/kafka-log-deletion-and-load-balancing-across-consumers
"
Since Kafka 0.8 already maintains a client offset, i would like to request
a feature, where a single partition consumption can be round robin across
11 matches
Mail list logo