On Tue, Mar 6, 2018, at 1:28 PM, Lance Bragstad wrote:
> Hey all,
> Last week during the PTG the keystone team sat down with a few of the
> infra folks to discuss performance testing. The major hurdle here has
> always been having dedicated hosts to use for performance testing,
> regardless of that being rally, tempest, or a home-grown script.
> Otherwise results vary wildly from run to run in the gate due to
> differences from providers or noisy neighbor problems.
> Opening up the discussion here because it sounded like some providers
> (mnaser, mtreinish) had some thoughts on how we can reserve specific
> hardware for these cases.
Currently the Infra team has access to a variety of clouds, but due to how
scheduling works we can't rule out noisy neighbors (or even being our own noisy
neighbor). mtreinish also has data showing that runtimes are too noisy to do
statistical analysis on, even within a single cloud region. So this is indeed
an issue in the current setup.
One approach that has been talked about in the past is to measure performance
impacting operations using metrics other than execution time. For example
number of sql queries or rabbit requests. I think this would also be valuable
but won't give you proper performance measurements.
That brought us back to the idea of possibly working with some cloud providers
like mnaser and/or mtreinish to have a small number of dedicated instances to
run performance tests on. We could then avoid the noisy neighbor problem as
For the infra team we would likely need to have at least two providers
providing these resources so that we could handle the loss of one without
backing up job queues. I don't think the hardware needs to have an other
special properties as we don't care about performance on specific hardware as
much as comparing performance of the project over time on known hardware.
Curious to hear what others may have to say.
OpenStack Development Mailing List (not for usage questions)