__
> From: John Coleman
> Sent: Friday, November 13, 2020 4:09 AM
> To: users@kafka.apache.org
> Subject: High latency for Kafka transactions
>
> Hi,
>
> We have very high latency ~1 second when we send a small number of
> small messages
13, 2020 4:09 AM
To: users@kafka.apache.org
Subject: High latency for Kafka transactions
Hi,
We have very high latency ~1 second when we send a small number of small
messages to Kafka. The whole group of messages is less than 1k. We are thinking
better to just send the messages
Hi,
We send small transactions (2 to 3 messages) a few times per second from each
of 5 to 10 producers, depending on the consumer partition size.
Regards,
John
On 14/11/20 03:55, "Guozhang Wang" wrote:
Hello John,
It's a bit hard to reason what's your total producing traffic to the
On 2020/11/13 09:09:00, John Coleman wrote:
> Hi,
>
> We have very high latency ~1 second when we send a small number of small
> messages to Kafka. The whole group of messages is less than 1k. We are
> thinking better to just send the messages together as 1 message containing a
>
to profile the app to see where the time is being spent.
[1]: https://kafka.apache.org/documentation/#linger.ms
From: John Coleman
Sent: Friday, November 13, 2020 4:09 AM
To: users@kafka.apache.org
Subject: High latency for Kafka transactions
Hi,
We have very high
Hello John,
It's a bit hard to reason what's your total producing traffic to the Kafka
brokers, e.g. how frequently do you send a `whole group of messages`? If it
is every 10ms e.g. then your traffic would be 100K per sec, which is pretty
large enough with transactions.
Guozhang
On Fri, Nov
Hi,
We have very high latency ~1 second when we send a small number of small
messages to Kafka. The whole group of messages is less than 1k. We are thinking
better to just send the messages together as 1 message containing a collection
of all the messages.
Is there something that we can tweak
Hi Gwen,
Well I have gone through this link while trying to setup my Logstash Kafka
handler,
https://github.com/joekiller/logstash-kafka
I could achieve what I was looking for but the performance is badly
affected while trying to write a big file of GB's.
I guess there should be some way so as
I am not familiar with logstash, but in custom log replay tool (used to
replay messages logged locally in case if e.g. kafka was not available and
useful in some other scenarios) I've seen it reaching 30,000 messages/sec
with avg message size of 4.5 kilobytes, all with regular production load on
Hi All,
I am having some log files of around 30GB, I am trying to event process
these logs by pushing them to Kafka. I could clearly see the throughput
achieved while publishing these event to Kafka is quiet slow.
So as mentioned for the single log file of 30GB, the Logstash is
continuously
I'm wondering how much of the time is spent by Logstash reading and
processing the log vs. time spent sending data to Kafka. Also, I'm not
familiar with log.stash internals, perhaps it can be tuned to send the data
to Kafka in larger batches?
At the moment its difficult to tell where is the
11 matches
Mail list logo
