The experiments you're doing are basically Monte Carlo discrete event
simulations - except that you're doing "hardware in the loop" instead of a
mathematical abstraction using a tool like ARENA, Simpy, etc. In each case
you have two choices:
Ignore the warm-up time and do a very long run (this will mask the
transient startup behavior). I sometimes do this when I'm pressed for
time or lazy.
Put in a warm-up time.
Figuring out the appropriate warm-up time is not trivial. In my case, I
wait for the system to become "wide-sense stationary"; that is, the mean
and variance stabilize. An alternative (favored by our local statistician)
is to look at the time series (we look at latency) and see when then the
time series appears to stabilize.
Wide-sense stationarity time can be estimated by using an online algorithm
such as Welford's algorithm; it's around a dozen lines of Python or C code
and doesn't require storing all data points (it is an unbiased estimator
and computes mean and variance on-the-fly). One then looks at the
difference between the current mean estimate and the previous mean estimate
and compares the differences to a predetermined reference (I use a relative
absolute value percent difference). This is somewhat subjective.
Both items 1 and 2 assume the system being measured doesn't change behavior
with respect to time; as long as the system is not running under heavy
load, this will probably be true.
There's at least one other approach I've used but that's really non-trivial
and is best left for another time.
Best regards,
Bo
Bohdan L. Bodnar
Lead Performance Engineer
1-312-871-5163
E-mail: [email protected] 222 South Riverside
Plaza
Chicago, IL
60606
United
States
From: Isuru Perera <[email protected]>
To: JMeter Users List <[email protected]>, Deepak Shetty
<[email protected]>
Date: 05/24/2017 01:37 AM
Subject: Re: Warming up Java servers before measuring performance
Hi Deepak,
Thanks a lot for the reply.
We are just using out of the box listeners for now.
I agree that the excluding the warmup time may hide the issues.
However, I believe that warming up the Java server is very important
especially when considering the JIT compilation in HotSpot JVM. JMH (
http://openjdk.java.net/projects/code-tools/jmh/) tool also warms up the
methods before taking measurements.
That's why I wanted to ask in this list whether other users have the same
requirements to exclude warmup time from the final results.
We can write a simple program to split the JTL, which will write warmup and
measurement requests to two different file. In this way, we don't
completely ignore the warmup time and we can compare the warmup results
with the actual measurement results.
Splitting the JTL seemed to be an additional work and we have to process
large files again after the tests. I was wondering why JMeter didn't
provide such a feature out of the box similar to JMH.
Thanks!
Best Regards,
On Fri, May 19, 2017 at 11:37 PM, Deepak Shetty <[email protected]> wrote:
> Hi
> Your reporting engine of choice will usually support this filter. For
most
> non trivial load tests you have to analyze the data in either a DB or
> something like graphite etc - Almost all of which will trivially allow
you
> to filter out time frames you dont want.
> For out of box listeners , yes you could write a simple program to filter
> values from JTL.
>
> Ive not found JMeter *warmup* to be a significant issue to matter in most
> normal load tests (i.e. those that run for atleast an hour). The
> application itself is a different matter but thats very application
> specific.(And sometimes *warmup* hides issues)
>
> On Wed, May 17, 2017 at 11:48 PM, Isuru Perera <[email protected]>
> wrote:
>
> > Hi,
> >
> > We are using JMeter to run load tests on our Java servers and we want
to
> > warm-up the application before taking measurements.
> >
> > For example, I want to have 10 minutes as the warm up time and 20
minutes
> > as the measurement time.
> >
> > I see two approaches to this.
> >
> > 1. Run JMeter for 30 minutes and exclude first 10 minutes data in the
JTL
> > file and generate reports.
> > 2. Run JMeter for 10 minutes and run again for 20 minutes. Generate
> reports
> > from 20 minutes JTL file.
> >
> > We prefer number 1, as the Java server will be in a steady state when
> > taking measurements. However for approach 1, we need to find a way to
> > exclude first few minutes of data. We can do this with a simple
program.
> >
> > Number 2 approach may not be useful as we are restarting the JMeter and
> it
> > will take some time get a steady throughput.
> >
> > I searched the web and I found following Stackoverflow question.
> > http://stackoverflow.com/questions/32237837/how-to-
> > exclude-warmup-time-from-jmeter-summary
> >
> > With GraphsGeneratorListener
> > <http://jmeter-plugins.org/wiki/GraphsGeneratorListener/>, it seems
> that I
> > can only generate reports. Is there any other way to just exclude first
> few
> > minutes in the JTL and then use the JTL to generate reports (for
example,
> > JMeter Dashboard). Is that listener the only standard way to exclude
> > warm-up time in JMeter?
> >
> > Why don't I see anything mentioned about Java Warm-up time in official
> > docs? Since JMeter is based on Java, I expected some guidelines to load
> > test Java servers and how to exclude Java warm-up time. I think we
should
> > also consider the warm-up time of JMeter. With approach 1, we can also
> make
> > sure that JMeter is also performing at its best when taking the
> > measurements.
> >
> > I really appreciate your feedback on this.
> >
> > Thank you.
> >
> > Best Regards,
> > --
> > Isuru Perera
> > about.me/chrishantha
> >
>
--
Isuru Perera
about.me/chrishantha
