On 9/10/14, 3:58 PM, "Devananda van der Veen" <devananda....@gmail.com> wrote:
>I'm going to assume that, for these benchmarks, you configured all the >services optimally. Sorry for any confusion; I am not trying to hide anything about the setup. I thought I was pretty transparent about the way uWSGI, MongoDB, and Redis were configured. I tried to stick to mostly default settings to keep things simple, making it easier for others to reproduce/verify the results. Is there further information about the setup that you were curious about that I could provide? Was there a particular optimization that you didn’t see that you would recommend? >I'm not going to question why you didn't run tests >with tens or hundreds of concurrent clients, If you review the different tests, you will note that a couple of them used at least 100 workers. That being said, I think we ought to try higher loads in future rounds of testing. >or why you only ran the >tests for 10 seconds. In Round 1 I did mention that i wanted to do a followup with a longer duration. However, as I alluded to in the preamble for Round 2, I kept things the same for the redis tests to compare with the mongo ones done previously. We’ll increase the duration in the next round of testing. >Instead, I'm actually going to question how it is that, even with >relatively beefy dedicated hardware (128 GB RAM in your storage >nodes), Zaqar peaked at around 1,200 messages per second. I went back and ran some of the tests and never saw memory go over ~20M (as observed with redis-top) so these same results should be obtainable on a box with a lot less RAM. Furthermore, the tests only used 1 CPU on the Redis host, so again, similar results should be achievable on a much more modest box. FWIW, I went back and ran a couple scenarios to get some more data points. First, I did one with 50 producers and 50 observers. In that case, the single CPU on which the OS scheduled the Redis process peaked at 30%. The second test I did was with 50 producers + 5 observers + 50 consumers (which claim messages and delete them rather than simply page through them). This time, Redis used 78% of its CPU. I suppose this should not be surprising because the consumers do a lot more work than the observers. Meanwhile, load on the web head was fairly high; around 80% for all 20 CPUs. This tells me that python and/or uWSGI are working pretty hard to serve these requests, and there may be some opportunities to optimize that layer. I suspect there are also some opportunities to reduce the number of Redis operations and roundtrips required to claim a batch of messages. The other thing to consider is that in these first two rounds I did not test increasing amounts of load (number of clients performing concurrent requests) and graph that against latency and throughput. Out of curiosity, I just now did a quick test to compare the messages enqueued with 50 producers + 5 observers + 50 consumers vs. adding another 50 producer clients and found that the producers were able to post 2,181 messages per second while giving up only 0.3 ms. --KG _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
