[openstack-dev] [zaqar] Juno Performance Testing (Round 3)

2014-09-16 Thread Kurt Griffiths
Hi crew, as promised I’ve continued to work through the performance test
plan. I’ve started a wiki page for the next batch of tests and results:

https://wiki.openstack.org/wiki/Zaqar/Performance/PubSub/Redis

I am currently running the same tests again with 2x web heads, and will
update the wiki page with them when they finish (it takes a couple hours
to run each batch of tests). Then, I plan to add an additional Redis proc
and run everything again. After that, there are a few other things that we
could do, depending on what everyone wants to see next.

* Run all these tests for producer-consumer (task distribution) workloads
* Tune Redis, uWSGI and see if we can't improves latency, stdev, etc.
* Do a few runs with varying message sizes
* Continue increasing load and adding additional web heads
* Continue increasing load and adding additional redis procs
* Vary number of queues
* Vary number of project-ids
* Vary message batch sizes on post/get/claim

P.S. As mentioned in yesterday’s team meeting, we are starting to work
on integration with Rally as our long-term performance testing solution. I
encourage everyone to get involved and contribute to that project; your
contributions benefit not only Zaqar, but other OS projects as well who
are starting to use Rally to keep an eye on their own services’
performance. Kudos to Boris et al. for all their hard work there.

--Kurt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [zaqar] Juno Performance Testing (Round 3)

2014-09-16 Thread Jay Pipes

On 09/16/2014 11:23 AM, Kurt Griffiths wrote:

Hi crew, as promised I’ve continued to work through the performance test
plan. I’ve started a wiki page for the next batch of tests and results:

https://wiki.openstack.org/wiki/Zaqar/Performance/PubSub/Redis

I am currently running the same tests again with 2x web heads, and will
update the wiki page with them when they finish (it takes a couple hours
to run each batch of tests). Then, I plan to add an additional Redis proc
and run everything again. After that, there are a few other things that we
could do, depending on what everyone wants to see next.

* Run all these tests for producer-consumer (task distribution) workloads
* Tune Redis, uWSGI and see if we can't improves latency, stdev, etc.
* Do a few runs with varying message sizes
* Continue increasing load and adding additional web heads
* Continue increasing load and adding additional redis procs
* Vary number of queues
* Vary number of project-ids
* Vary message batch sizes on post/get/claim


Don't forget my request to identify the effect of keepalive settings in 
uWSGI :)


Thanks!
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [zaqar] Juno Performance Testing (Round 3)

2014-09-16 Thread Kurt Griffiths
Thanks for the reminder! I’ll make note of that. In these tests the
clients are hitting Nginx (which is acting as a load balancer) so I could
try disabling keep-alive there and seeing what happens. So far I just used
the default that was written into the conf when the package was installed
(keepalive_timeout 65;”).

FWIW, I started an etherpad to track a shortlist of ideas:

https://etherpad.openstack.org/p/zaqar-performance-testing


On 9/16/14, 11:58 AM, Jay Pipes jaypi...@gmail.com wrote:

On 09/16/2014 11:23 AM, Kurt Griffiths wrote:
 Hi crew, as promised I’ve continued to work through the performance test
 plan. I’ve started a wiki page for the next batch of tests and results:

 https://wiki.openstack.org/wiki/Zaqar/Performance/PubSub/Redis

 I am currently running the same tests again with 2x web heads, and will
 update the wiki page with them when they finish (it takes a couple hours
 to run each batch of tests). Then, I plan to add an additional Redis
proc
 and run everything again. After that, there are a few other things that
we
 could do, depending on what everyone wants to see next.

 * Run all these tests for producer-consumer (task distribution)
workloads
 * Tune Redis, uWSGI and see if we can't improves latency, stdev, etc.
 * Do a few runs with varying message sizes
 * Continue increasing load and adding additional web heads
 * Continue increasing load and adding additional redis procs
 * Vary number of queues
 * Vary number of project-ids
 * Vary message batch sizes on post/get/claim

Don't forget my request to identify the effect of keepalive settings in
uWSGI :)

Thanks!
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [zaqar] Juno Performance Testing (Round 3)

2014-09-16 Thread Kurt Griffiths
All results have been posted for the 2x web head + 2x redis tests (C3). I
also rearranged the wiki page to make it easier to compare the impact of
adding more capacity on the backend.

In C3, The load generator’s CPUs were fully saturated, while there was
still plenty of headroom on the web heads (~60% remaining CPU capacity)
and redis server instances (50% remaining across 2 procs), based on load
average. Therefore, I think this setup could actually handle a lot more
load than I attempted, and that would be a good followup test.

Keeping in mind that I haven’t spent much time tuning the systems and the
Redis driver still has several opportunities for further optimization, I
think these results so far are a pretty good sign we are on the right
track. I was happy to see that all requests were succeeding and I didn’t
find any race conditions during the tests, so kudos to the team for their
careful coding and reviews!

On 9/16/14, 2:43 PM, Kurt Griffiths kurt.griffi...@rackspace.com wrote:

Results are now posted for all workloads for 2x web heads and 1x Redis
proc (Configuration 2). Stats are also available for the write-heavy
workload with 2x webheads and 2x redis procs (Configuration 3). The latter
results look promising, and I suspect the setup could easily handle a
significantly higher load. Multiple load generator boxes would need to be
used (noted on the etherpad).

On 9/16/14, 10:23 AM, Kurt Griffiths kurt.griffi...@rackspace.com
wrote:

Hi crew, as promised I’ve continued to work through the performance test
plan. I’ve started a wiki page for the next batch of tests and results:

https://wiki.openstack.org/wiki/Zaqar/Performance/PubSub/Redis

I am currently running the same tests again with 2x web heads, and will
update the wiki page with them when they finish (it takes a couple hours
to run each batch of tests). Then, I plan to add an additional Redis proc
and run everything again.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev