On 10/18/2013 12:17 PM, Boris Pavlovic wrote:
John,
Actually seems like a pretty good suggestion IMO, at least something
worth some investigation and consideration before quickly discounting
it. Rather than "that's not what tempest is", maybe it's something
tempest "could do". Don't know, not saying one way or the other, just
wondering if it's worth some investigation or thought.
These investigations I made before start working around Rally. It was
about 3 months ago.
It is not "quickly discounting" I didn't have yesterday time to make
long response, so I will write it today:
I really don't like to make a copy of another projects, so I tried to
reuse all projects & libs that we already have.
To explain why we shouldn't merge Rally & Tempest in one project (and
should keep both) we should analyze their use cases.
1. DevStack - one "click" and get your OpenStack cloud from sources
2. Tempest - one "click" and get your OpenStack Cloud verified
Both of these projects are great, because they are very useful and solve
complicated tasks without "pain" for end user. (and I like them)
3. Rally is also one "click" system that solve OpenStack benchmarking.
To clarify situation. We should analyze what I mean by one "click"
benchmarking and what are the use cases.
Use Cases:
1. Investigate how deployments influence on OS performance (find the set
of good OpenStack deployment architectures)
2. Automatically get numbers & profiling info about how your changes
influence on OS performance
3. Using Rally profiler detect scale & performance issues.
Like here when we are trying to delete 3 VMs by one request they are
deleted one by one because of DB lock on quotas table
http://37.58.79.43:8080/traces/0011f252c9d98e31
4. Determine maximal load that could handle production cloud
To cover these cases we should actually test OpenStack deployments
making simultaneously OpenStack API calls.
So to get results we have to:
1. Deploy OpenStack cloud somewhere. (Or get existing cloud)
2. Verify It
3. Run Benchmarks
4. Collect all results & present it in human readable form.
The goal of Rally was designed to automate these steps:
1.a Use existing cloud.
1.b.1 Automatically get (virtual) Servers from (soft layer, Amazon,
RackSpace or you private public cloud, or OpenStack cloud)
1.b.2 DeployOpenStack on these servers from source (using Devstack,
Anvli, Fuel or TrippleO...).
1.b.3 Patch this OpenStack with tomograph to get profiling information
(I hope we will merge these patches into upstream).
2. Using tempest verify this cloud (we are going to switch from
fuel-ostf-tests)
3. Run specified parametrized (to be able to make different load)
benchmark scenarios
4. Collect all information about execution & present it in human
readable form. (Tomograph, Zipking, matplotlib...)
Honestly, there has been so much work done in Tempest with the stress
tests and the scenario tests in this release, and a large growing
community around those, that it doesn't make any sense to me that you
put item #3 as a non Tempest thing.
It feels very "not invented here".
Are you guys doing a summit session somewhere on this.
It also feels like the efforts around #4 would be much better served in
the OpenStack community if they were integrated around testr and subunit
so they could be reused in many contexts.
I also think 1.b.3 is probably better done in the way the coverage
extension was done for nova, something which is baked in and can be
administratively turned on, not something which requires a hot patch to
the system.
It's cool to have performance analysis tooling, but if it arrives in a
way that doesn't integrate well with the rest of OpenStack, the impact
is going to be far less than it could be. I'd like us to get the full
bang for our buck out of efforts like this, especially if there is hope
for it to graduate from stackforge into one of our standard toolkits.
-Sean
--
Sean Dague
http://dague.net
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
