How to avoid Kafka latency spikes caused by log segment flush

ckOverflow (https://stackoverflow.com/questions/73555649/how-to-avoid-kafka-latency-spikes-caused-by-log-segment-flush) and I'm really trying to get help. Thanks, Jiri

Re: kafka latency for large message

that's very good information from the slides, thanks. Our design to use kafka has 2 purpose. one is use it as a cache, we use ktable for that purpose, second purpose is use as message delivery mechanism to send it to other system. Because we very much care the latency, the ktable with a compact

Re: kafka latency for large message

Hi Nan, Would you consider other approaches that may actually be a more efficient solution for you? There is a slide deck Handle Large Messages In Apache Kafka . For messages this large, one of the approaches

Re: kafka latency for large message

It takes time to send that much data over the network. Why would you expect a smaller latency? On Mon, Mar 18, 2019 at 8:05 AM Nan Xu wrote: > anyone can give some suggestion? or an explanation why kafka give a big > latency for large payload. > > Thanks, > Nan > > On Thu, Mar 14, 2019 at 3:53

Re: kafka latency for large message

anyone can give some suggestion? or an explanation why kafka give a big latency for large payload. Thanks, Nan On Thu, Mar 14, 2019 at 3:53 PM Xu, Nan wrote: > Hi, > > We are using kafka to send messages and there is less than 1% of > message is very big, close to 30M. understanding kafka

kafka latency for large message

Hi, We are using kafka to send messages and there is less than 1% of message is very big, close to 30M. understanding kafka is not ideal for sending big messages, because the large message rate is very low, we just want let kafka do it anyway. But still want to get a reasonable

Kafka latency optimization

Hi, everyone My kafka version is 0.10.2.1. My service have really low qps (1msg/sec). And our requirement for rtt is really strict. ( 99.9% < 30ms) Currently I've encounter a problem, when kafka run for a long time, 15 days or so, performance start to go down. 2017-10-21 was like Time .

Re: kafka latency

Hello Chao, How did you measure latency? See also https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines Kind regards, Stevo Slavic. On Tue, Jul 26, 2016 at 9:52 PM, Luo, Chao wrote: > Dear Kafka guys, > > I measured

kafka latency

Dear Kafka guys, I measured the latency of kafka system. The producer, kafka servers, and consumer were running on different machines (AWS EC2 instances). The producer-to-consumer latency is around 250 milliseconds. Is it a normal value for kafka system. Can I do better and how? Any comments

Re: Trying to figure out kafka latency issues

Kurian [ra...@signalfuse.com] Received: Sunday, 21 Dec 2014, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab

Re: Trying to figure out kafka latency issues

To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine

Re: Trying to figure out kafka latency issues

help there. Good luck! -Original Message- From: Rajiv Kurian [ra...@signalfuse.com] Received: Sunday, 21 Dec 2014, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency

Re: Trying to figure out kafka latency issues

! -Original Message- From: Rajiv Kurian [ra...@signalfuse.com] Received: Sunday, 21 Dec 2014, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile

Re: Trying to figure out kafka latency issues

[users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine

Re: Trying to figure out kafka latency issues

to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine not JVM heap) free disk space on the broker. I'll need to take a look at the JVM metrics too. What seemed

Re: Trying to figure out kafka latency issues

] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine not JVM heap) free disk space on the broker. I'll need to take a look at the JVM

Re: Trying to figure out kafka latency issues

: Sunday, 21 Dec 2014, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine

Re: Trying to figure out kafka latency issues

luck! -Original Message- From: Rajiv Kurian [ra...@signalfuse.com] Received: Sunday, 21 Dec 2014, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right

Re: Trying to figure out kafka latency issues

, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine not JVM heap

Re: Trying to figure out kafka latency issues

: Sunday, 21 Dec 2014, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory

Re: Trying to figure out kafka latency issues

out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine not JVM heap) free disk space on the broker. I'll need to take a look at the JVM

Re: Trying to figure out kafka latency issues

with the Java client so can't really help there. Good luck! -Original Message- From: Rajiv Kurian [ra...@signalfuse.com] Received: Sunday, 21 Dec 2014, 12:25PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC

RE: Trying to figure out kafka latency issues

@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues Some more work tells me that the end to end latency numbers vary with the number of partitions I am writing to. I did an experiment, where based on a run time flag I would dynamically select how many of the *1024

Re: Trying to figure out kafka latency issues

-Original Message- From: Rajiv Kurian [ra...@signalfuse.com] Received: Saturday, 20 Dec 2014, 10:24PM To: users@kafka.apache.org [users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues Some more work tells me that the end to end latency numbers vary with the number

RE: Trying to figure out kafka latency issues

[users@kafka.apache.org] Subject: Re: Trying to figure out kafka latency issues I'll take a look at the GC profile of the brokers Right now I keep a tab on the CPU, Messages in, Bytes in, Bytes out, free memory (on the machine not JVM heap) free disk space on the broker. I'll need to take a look

Trying to figure out kafka latency issues

I am trying to replace a Thrift peer to peer API with kafka for a particular work flow. I am finding the 99th percentile latency to be unacceptable at this time. This entire work load runs in an Amazon VPC. I'd greatly appreciate it if some one has any insights on why I am seeing such poor

RE: Trying to figure out kafka latency issues

@kafka.apache.org] Subject: Trying to figure out kafka latency issues I am trying to replace a Thrift peer to peer API with kafka for a particular work flow. I am finding the 99th percentile latency to be unacceptable at this time. This entire work load runs in an Amazon VPC. I'd greatly appreciate it if some one

Re: Trying to figure out kafka latency issues

On Sat, Dec 20, 2014 at 3:49 PM, Rajiv Kurian ra...@signalfuse.com wrote: I am trying to replace a Thrift peer to peer API with kafka for a particular work flow. I am finding the 99th percentile latency to be unacceptable at this time. This entire work load runs in an Amazon VPC. I'd greatly

Re: Trying to figure out kafka latency issues

I've done some more measurements. I've also started measuring the latency between when I ask my producer to send a message and when I get an acknowledgement via the callback. Here is my code: // This function is called on every producer once every 30 seconds. public void addLagMarkers(final

Re: Trying to figure out kafka latency issues

Some more work tells me that the end to end latency numbers vary with the number of partitions I am writing to. I did an experiment, where based on a run time flag I would dynamically select how many of the *1024 partitions* I write to. So say I decide I'll write to at most 256 partitions I mod

Re: Kafka latency measures

There were actually several patches against trunk since 0.8.1.1 that may impact latency however, especially when using acks=-1. So those results in the blog may be a bit better than what you would see in 0.8.1.1. -Jay On Wed, Jun 18, 2014 at 7:58 PM, Supun Kamburugamuva supu...@gmail.com wrote:

Kafka latency measures

Hi, We are trying to evaluate Kafka for a real time application. We are sending 50 Kb messages at a fixed rate. The normal messages have a reasonable latency. But then there are these outliers that takes unpredictable amount of time. This causes the average latency to increase dramatically. We

Re: Kafka latency measures

Hi, do these spikes happen to correlate with log.flush.interval.messages or log.flush.interval.ms? If so it's the file system sync blockage you are seeing. /Magnus 2014-06-18 16:31 GMT+02:00 Supun Kamburugamuva supu...@gmail.com: Hi, We are trying to evaluate Kafka for a real time

Re: Kafka latency measures

Which version of Kafka did you use? When you say latency, do you mean the latency between the producer and consumer? If so, are you using a timestamp within the message to compute this latency? On Wed, Jun 18, 2014 at 8:15 AM, Magnus Edenhill mag...@edenhill.se wrote: Hi, do these spikes

Re: Kafka latency measures

The spikes happens without any correlation with the log.flush.interval.message. They happen more frequently. I'm using the latest version. I'm sending the messages to Kafka, then there is a message receiver, it sends the same messages back through kafka to original sender. The round trip latency

Re: Kafka latency measures

what are the latency numbers you observed, avg as well as worst case? Here is a blog that we did recently which should reflect latest performance metrics for latency - http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines On Wed, Jun 18,

Re: Kafka latency measures

My machine configuration is not very high. The average one way latency we observe is around 10 ~ 15 ms for 50k messages. The outliers doesn't occur for small messages. For small messages we observe around 6 ms latency. Thanks, Supun.. On Wed, Jun 18, 2014 at 10:18 PM, Neha Narkhede