Re: HDD or SSD or EBS for kafka brokers in Amazon EC2

Henry: We run Kafka on the old and trusty m1.xlarge. We avoid EBS completely, it's network storage that pretends to be local and when the network, which is AWS' weak spot, acts up EBS is a big liability. It's also slow and expensive. Others: Thanks for sharing your experience with the d2's. We

Re: Is fetching from in-sync replicas possible?

Thank you for your response Joel, Can you file a jira for this? I've created https://issues.apache.org/jira/browse/KAFKA-2225 Out of curiosity - what's the typical latency (distribution) you see between zones? Unfortunately I don't have any good numbers on that. Since we're publishing both

Is fetching from in-sync replicas possible?

Hello, Apologies if this question has been asked before. If I understand things correctly a client can only fetch from the leader of a partition, not from an (in-sync) replica. I have a use case where it would be very beneficial if it were possible to fetch from a replica instead of just the

Re: Problems decoding fetch responses, getting messages that are bigger than their message sets

remainingBufferLength. Regards, Magnus 2015-03-01 8:55 GMT+01:00 Theo Hultberg t...@iconara.net: That the message set size was always the same when this happened made me start looking for that number, and it turned out that was what I had set as the MaxBytes field of the requests. This make me

Re: Problems decoding fetch responses, getting messages that are bigger than their message sets

, Theo On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg t...@iconara.net wrote: Hi, Mostly out of curiosity I'm writing a Kafka client, and I'm getting stuck at decoding fetch responses. Most of the time everything goes fine, but quite often I get frames back that seem to be wrong. I'm sure

Problems decoding fetch responses, getting messages that are bigger than their message sets

Hi, Mostly out of curiosity I'm writing a Kafka client, and I'm getting stuck at decoding fetch responses. Most of the time everything goes fine, but quite often I get frames back that seem to be wrong. I'm sure I've misunderstood something about the spec, so maybe someone can get me on the right

Re: Problems decoding fetch responses, getting messages that are bigger than their message sets

in the protocol guide. Is what I have found the thing described as As an optimization the server is allowed to return a partial message at the end of the message set. Clients should handle this case.? yours, Theo On Sat, Feb 28, 2015 at 10:19 PM, Theo Hultberg t...@iconara.net wrote: Here's an example

Re: Replication bandwidth double what is expected

the wire. Guozhang On Mon, Sep 1, 2014 at 11:06 PM, Theo Hultberg t...@iconara.net wrote: Hi Guozhang, We're using the default on all of those, except num.replica.fetchers which is set to 4. T# On Mon, Sep 1, 2014 at 9:41 PM, Guozhang Wang wangg...@gmail.com wrote

Re: Replication bandwidth double what is expected

, replica.fetch.wait.max.ms and num.replica.fetchers configs? Guozhang On Mon, Sep 1, 2014 at 2:52 AM, Theo Hultberg t...@iconara.net wrote: Hi, We're evaluating Kafka, and have a problem with it using more bandwidth than we can explain. From what we can tell the replication uses at least twice

Replication bandwidth double what is expected

Hi, We're evaluating Kafka, and have a problem with it using more bandwidth than we can explain. From what we can tell the replication uses at least twice the bandwidth it should. We have four producer nodes and three broker nodes. We have enabled 3x replication, so each node will get a copy of