Thank you for the KIP, Matt. Totally agree on having a warm-up for benchmark testing. The initial producer setup time could involve things such as network connection setup (including authN, SSL handshake etc), DNS resolution, metadata fetching etc which could impact the result of steady-state performance.
May I suggest adding some more clarity to the KIP about the following: 1. Should we also include the metrics for warm-up separately (instead of having them as 0)? This would have the advantage of reporting both warm up performance and steady state performance in the same benchmark run. A similar report is followed by JMH (https://github.com/openjdk/jmh) as well. You can look at it for some inspiration. 2. Please add validation that num-records should be greater than warm-up records. Else report an error. 3. Please add a recommendation in the docs for the tool on what an ideal value for warm up should be. For users who may not be completely familiar with producer buffering / back-pressure, it would be useful to understand a good value to set. In my opinion, 4. I wonder how the --throughput parameter works with the warmup! Could we have a situation where the "steady-state" is impacted by the warm-up traffic? As an example, we could land in a situation where the slow processing of warm-up messages could impact the measurement of steady-state. This could happen in a situation when warm-up messages are waiting to be processed on the server (or maybe on the producer buffer) but we have started recording end-to-end latency for the steady-state messages. I imagine this should be ok because it achieves the purpose of removing bootstrap times, but I haven't been able to reason about it in my head. What are your thoughts on this? -- Divij Vaidya On Fri, Jun 14, 2024 at 12:23 AM Eric Lu <erickandmorty2...@gmail.com> wrote: > Hi Matt, > > Yes I forgot to update the KIP counter after creating a KIP. I changed mine > to 1053. We should be all good now. > > Cheers, > Eric > > On Thu, Jun 13, 2024 at 3:08 PM Welch, Matt <matt.we...@intel.com> wrote: > > > Hello again Kafka devs, > > > > I'd like to again call attention to this KIP for discussion. > > Apparently, we encountered a race condition when choosing KIP numbers, > but > > hopefully it's straightened out now. > > > > Regards, > > Matt > > > > > > -----Original Message----- > > From: Welch, Matt <matt.we...@intel.com> > > Sent: Thursday, June 6, 2024 4:44 PM > > To: dev@kafka.apache.org > > Subject: [DISCUSS] KIP-1052: Enable warmup in producer performance test > > > > Hello all, > > > > I'd like to propose a change that would allow the producer performance > > test to have a warmup phase where the statistics gathered could be > > separated from statistics gathered during steady state. > > > > Although startup is an important phase of Kafka operations and special > > attention should be paid to optimizing startup performance, often we > would > > like to understand Kafka performance during steady-state operation, > > separate from its performance during producer startup. It's common for > new > > producers, like in a fresh producer performance test run, to have high > > latency during startup. This high latency can complicate the > understanding > > of steady-state performance, even when collecting long-running tests. If > > we want to understand steady-state latency separate from startup latency, > > we can collect measurements for each phase in disjoint sets then present > > statistics on each set independently or as a combined population of > > measurements. This feature would be completely optional and could be > > represented by a new command line flag for the producer performance test, > > '--warmup-records'. > > > > KIP: KIP-1052: Enable warmup in producer performance test - Apache Kafka > - > > Apache Software Foundation< > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1052%3A+Enable+warmup+in+producer+performance+test > > > > > > > Thank you, > > Matt Welch > > > > >
