>
> Date: Wed, 06 Aug 2014 09:44:12 -0400
> From: Sean Dague <s...@dague.net>
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] Which program for Rally
> Message-ID: <53e2312c.8000...@dague.net>
> Content-Type: text/plain; charset=utf-8
>
> Like the fact that right now the rally team is proposing gate jobs which
> have some overlap to the existing largeops jobs. Did they start a
> conversation about it? Nope. They just went off to do their thing
> instead. https://review.openstack.org/#/c/112251/
>

Hi Sean,

Appreciate your analysis
Here is a comparison of the tempest largeops job and similar in Rally.

What large-ops job provides as of now:
Running hard-coded configured benchmarks (in gates), that are taken from
tempest repo.
eg:  "run 100 vms by one request". End result is a +1 or -1 which doesnt
really reflect much in terms of the performance stats and regressions in
performance.


What Rally job provides:
(example in glance:
https://github.com/openstack/glance/tree/master/rally-scenarios)

1) Projects can specify which benchmarks to run:
https://github.com/openstack/glance/blob/master/rally-scenarios/glance.yaml

2) Projects can specify conditions of passing and inputs to benchmarks
(e.g. there is no iteration of benchmark failed and average duration of
iteration is less then X)
https://github.com/stackforge/rally/blob/master/rally-scenarios/rally.yaml#L11-L12


3) Projects can create any number of benchmarks inside their source tree
(so they don't need to merge anything to rally)
https://github.com/openstack/glance/tree/master/rally-scenarios/plugins

4) Users are getting automated reports of all benchmarks:
http://logs.openstack.org/81/112181/2/check/gate-rally-dsvm-rally/78b1146/rally-plot/results.html.gz

5) Users can easily install Rally (with this script
https://github.com/stackforge/rally/blob/master/install_rally.sh)
and test benchmark locally, using the same benchmark configuration as in
gate.

 6) Rally jobs (benchmarks) give you capabilities to check for SLAs in your
gates itself, which helps immensly to gauge impact of your proposed change
on the current code in terms of performance and SLA

Basically with Rally job, one can benchmark changes and compare them with
master in gates.
Using below approach:

1) Put patch set 1 that changes rally benchmark configuration and probably
adds some benchmark
Get base results

2) Put patch set2 that includes point 1 + changes that fixes issue
Get new results

3) Compare results and if new results are better push patch set 3 that
removes changes in task and merge it.



>
> So now we're going to run 2 jobs that do very similar things, with
> different teams adjusting the test loads. Which I think is basically
> madness.
>
>         -Sean
>
> --
> Sean Dague
> http://dague.net
>

Rally jobs allows every project to chose what benchmarks to have (as a
plugins in their source tree) & run in gates.

Rally is trying to be as open as possible by helping projects define and
set their own benchmarks in their gates which they have full control over.

I think this is very important point as it simplifies a lot work on
performance issues. Hopefully we can discuss out these issues in IRC or
someplace so that we are all on same page in terms of details about what
Rally does and what it doesnt do.

Rohan Kanade
Senior Software Engineer, Red Hat
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to