Apart from the actual request, the consumer sends metadata requests as
well. What is actually going through the network when you get these
Try increasing the metadata max age with metadata.max.age.ms.
(Disclaimer: I myself never ran any benchmarking, just trying to help :). )
On Mon, Aug 7, 2017 at 7:37 PM, Chao Wang <chaow...@wustl.edu> wrote:
> Thanks, David. I was trying to do Kafka pub/sub on a local, closed
> network. In my case, I observed microsecond-latency with bare-bone sockets,
> and I would like to know how to configure Kafka to achieve similar result;
> if it turned out to be infeasible, what might be the cause of the
> additional latency?
> On 08/07/2017 12:10 PM, David Garcia wrote:
>> You are not going to get that kind of latency (i.e. less than 100
>> microseconds). In my experience, consumer->producer latency averages
>> around: 20 milliseconds (cluster is in AWS with enhanced networking).
>> On 8/3/17, 2:32 PM, "Chao Wang" <chaow...@wustl.edu> wrote:
>> I observed that it took 2-6 milliseconds for a topic to be
>> received by a
>> Kafka consumer from a Kafka producer, and I wonder what I might be
>> missing or I was wrong in configuring Kafka for low latency
>> at < 100 microseconds). I did the following:
>> 1. On the broker, I tried to prevent frequent flush of data to
>> 2. On the producer, I tried to reduce the delay by setting
>> linger.ms=0, acks =0, and I invoked flush() right after send()
>> 3. On the consumer, I set poll(0) (i.e., fetch every data once
>> I got similar observation (millisecond latency) in varying
>> value size
>> from 1 to 512B, and also similar results when either colocating
>> producer/consumer or putting them on separate PCs (connecting by a
>> switch). As a verification, I implemented simple C/C++ sockets for
>> transmission and observed latencies no more than 100 microseconds.