Re: High latency for Kafka transactions

[John Coleman]

Hi,

We are using Confluent.Kafka (dotnet) and according to the docs the linger.ms 
is default to 0.5ms which would cover a lot of serializations since each occurs 
in < 10 microseconds. A typical transaction batch would be 2 to 3 messages, 
occasionally more.

Regards,
John

On 14/11/20 05:24, "Gary Russell" <gruss...@vmware.com> wrote:

    Have you increased the producer linger.ms property [1] to ~1s? (Default is 
0 so records are sent immediately; it generally shouldn't be more than a few ms 
- depending on how long your serialization takes).

    If not, perhaps your serialization (or compression) might be the problem; 
best to profile the app to see where the time is being spent.

    [1]: 
https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkafka.apache.org%2Fdocumentation%2F%23linger.ms&amp;data=04%7C01%7CJohn.co%40central.tech%7Cee6e35911f484f03184208d88822d5e0%7C817e531d191b4cf58812f0061d89b53d%7C0%7C0%7C637409030504161229%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=t6XTDwN4QQs4kL5pMZzCSpcbjwMX95QrA8dzpDdrjyo%3D&amp;reserved=0
    ________________________________
    From: John Coleman <john...@central.tech>
    Sent: Friday, November 13, 2020 4:09 AM
    To: users@kafka.apache.org <users@kafka.apache.org>
    Subject: High latency for Kafka transactions

    Hi,

    We have very high latency ~1 second when we send a small number of small 
messages to Kafka. The whole group of messages is less than 1k. We are thinking 
better to just send the messages together as 1 message containing a collection 
of all the messages.

    Is there something that we can tweak to reduce this latency? We have 10 
producers and they publish to a topic with 5 consumers. Be nice if we don’t 
have to change our message structure.

    TIA
    John