Re: Kafka delay

2017-01-13 Thread Hans Jespersen
batch.size, linger.ms, max.in.flight.requests.per.connection, etc all effect the trade off between latency and throughput. Your throughput is very low so you should be able to tune for much better latency. Very easy to get 1-2 ms of latency out of Kafka. Here is a video on Tuning Kafka for low

Re: Kafka delay

2017-01-13 Thread Robert Quinlivan
Can you clarify what you mean by "process time"? Is this the time it takes to have a message sent from the producer appear in a consumer? You may be seeing the result of batching. The producer batches messages so if you are testing sending a single message at a time you will get different

Kafka delay

2017-01-13 Thread Patricia Callejo
Hi, we are developing a kafka environment in order to be able to process up hundreds of messages per second. The point is that if we send in different partitions a lot of messages, the time of the process is very good in ms, however if we send just a single message, the minimum time required