Re: Kafka delay

[Hans Jespersen]

batch.size, linger.ms, max.in.flight.requests.per.connection, etc all
effect the trade off between latency and throughput. Your throughput is
very low so you should be able to tune for much better latency. Very easy
to get 1-2 ms of latency out of Kafka.

Here is a video on Tuning Kafka for low latency guaranteed messaging --
Jiangjie (Becket) Qin (LinkedIn)
 which might give you much
better context and understanding of what these parameters mean and how they
work.

-hans

/**
 * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
 * h...@confluent.io (650)924-2670
 */

On Fri, Jan 13, 2017 at 12:36 AM, Patricia Callejo <
patricia.call...@imdea.org> wrote:

> Hi,
> we are developing a kafka environment in order to be able to process up
> hundreds of messages per second. The point is that if we send in different
> partitions a lot of messages, the time of the process is very good in ms,
> however if we send just a single message, the minimum time required is very
> high, at least 1 or 2 seconds, even more.
> Is there any configuration we are missing to avoid this minimum delay? or
> is the minimum time required by kafka for the whole process?
>
> Best Regards,
> Patricia

Re: Kafka delay

[Robert Quinlivan]

Can you clarify what you mean by "process time"? Is this the time it takes
to have a message sent from the producer appear in a consumer?

You may be seeing the result of batching. The producer batches messages so
if you are testing sending a single message at a time you will get
different performance characteristics than if you are sending many.

On Fri, Jan 13, 2017 at 2:36 AM, Patricia Callejo <
patricia.call...@imdea.org> wrote:

> Hi,
> we are developing a kafka environment in order to be able to process up
> hundreds of messages per second. The point is that if we send in different
> partitions a lot of messages, the time of the process is very good in ms,
> however if we send just a single message, the minimum time required is very
> high, at least 1 or 2 seconds, even more.
> Is there any configuration we are missing to avoid this minimum delay? or
> is the minimum time required by kafka for the whole process?
>
> Best Regards,
> Patricia




-- 
Robert Quinlivan
Software Engineer, Signal

Kafka delay

[Patricia Callejo]

Hi, 
we are developing a kafka environment in order to be able to process up 
hundreds of messages per second. The point is that if we send in different 
partitions a lot of messages, the time of the process is very good in ms, 
however if we send just a single message, the minimum time required is very 
high, at least 1 or 2 seconds, even more. 
Is there any configuration we are missing to avoid this minimum delay? or is 
the minimum time required by kafka for the whole process? 

Best Regards,
Patricia