Re: Benchmarking and stress testing Varnish
Those all seem very useful to me, but I think the lowest-hanging performance fruit right now is simultaneous connections and the threading model (including the discussions about stacksize and memory usage, etc). Modeling Varnish's behavior with certain ranges of simultaneous worker and backend threads would be useful, IMHO -- both from slow backends, slow clients, and lots of keepalive threads. -- kb On Nov 13, 2009, at 5:38 AM, Kristian Lyngstol wrote: > As some of you might already know, I regular stress tests of Varnish, most > of the time it's a matter of testing the same thing over and over to ensure > that there aren't any huge surprises during the development of Varnish (we > run nightly builds and stress tests of trunk - they need to be as > predictable as possible to have any value). > > However, I also do occasional tests which involve trying to find the > various breaking points of Varnish. Last time I did this, I used roughly 42 > cpu cores spread over about 30 computers to generate traffic against a > single Varnish server on our quad xenon core machine. The result thus far > was that all of the clients ran far too close to 100% cpu usage, and > Varnish was serving between 140 and 150 thousand requests per second. > > The reason I'm telling you this is because I'm looking for input on aspects > that should be tested next time I do this, which will most likely be during > Christmas (far easier to borrow machine power). So far on my list, I've > got: > > 1. Test performance over some time period when pages are evicted more > frequently. (ie: X thousand pages requested repeatedly, but random expiry > time). > > 2. Test with fewer requests per session (this is somewhat hard because the > clients tend to turn into the bottleneck). > > 3. Test raw hot-hit cache rate (more of what I did before - get a high > number). > > 4. Test raw performance with a huge data set that's bound to be swapped > out. > > 5. Various tests of -sfile versus -smalloc and large datasets, combined > with critbit/classic tests. > > 6. Find some reasonably optimal settings, then fidget around trying to find > a worst-case scenario. > > 7. Test the effect of session lingering with really slow clients. > > > > One thing that should be noted is that the test server is limited to 1gbit, > which means that for "raw req/s", we're basically forced to use tiny pages, > or we just end up starving the network. > > The goal is to test theories regarding performance, stability and > predictability. Basically find the breaking points, what's good and what's > not, and what we have to care about and what we can silently ignore. > > As you can see, the list is getting long and this is off the top of my > head, but any suggestions are welcome. > > -- > Kristian Lyngstøl > Redpill Linpro AS > Tlf: +47 21544179 > Mob: +47 99014497 > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Benchmarking and stress testing Varnish
Kristian, we do also test Varnish's behavior with slow clients, trying to find if it runs out of file descriptors, for instance. There was a time that when we reach 65k fd an assert In shmlog.c made it to bailout. I think it was fixed but not ported to 2.0.5. Regards, Rogerio Schneider Em 13/11/2009, às 11:38, Kristian Lyngstol escreveu: > As some of you might already know, I regular stress tests of > Varnish, most > of the time it's a matter of testing the same thing over and over to > ensure > that there aren't any huge surprises during the development of > Varnish (we > run nightly builds and stress tests of trunk - they need to be as > predictable as possible to have any value). > > However, I also do occasional tests which involve trying to find the > various breaking points of Varnish. Last time I did this, I used > roughly 42 > cpu cores spread over about 30 computers to generate traffic against a > single Varnish server on our quad xenon core machine. The result > thus far > was that all of the clients ran far too close to 100% cpu usage, and > Varnish was serving between 140 and 150 thousand requests per second. > > The reason I'm telling you this is because I'm looking for input on > aspects > that should be tested next time I do this, which will most likely be > during > Christmas (far easier to borrow machine power). So far on my list, > I've > got: > > 1. Test performance over some time period when pages are evicted more > frequently. (ie: X thousand pages requested repeatedly, but random > expiry > time). > > 2. Test with fewer requests per session (this is somewhat hard > because the > clients tend to turn into the bottleneck). > > 3. Test raw hot-hit cache rate (more of what I did before - get a high > number). > > 4. Test raw performance with a huge data set that's bound to be > swapped > out. > > 5. Various tests of -sfile versus -smalloc and large datasets, > combined > with critbit/classic tests. > > 6. Find some reasonably optimal settings, then fidget around trying > to find > a worst-case scenario. > > 7. Test the effect of session lingering with really slow clients. > > > > One thing that should be noted is that the test server is limited to > 1gbit, > which means that for "raw req/s", we're basically forced to use tiny > pages, > or we just end up starving the network. > > The goal is to test theories regarding performance, stability and > predictability. Basically find the breaking points, what's good and > what's > not, and what we have to care about and what we can silently ignore. > > As you can see, the list is getting long and this is off the top of my > head, but any suggestions are welcome. > > -- > Kristian Lyngstøl > Redpill Linpro AS > Tlf: +47 21544179 > Mob: +47 99014497 > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Benchmarking and stress testing Varnish
As some of you might already know, I regular stress tests of Varnish, most of the time it's a matter of testing the same thing over and over to ensure that there aren't any huge surprises during the development of Varnish (we run nightly builds and stress tests of trunk - they need to be as predictable as possible to have any value). However, I also do occasional tests which involve trying to find the various breaking points of Varnish. Last time I did this, I used roughly 42 cpu cores spread over about 30 computers to generate traffic against a single Varnish server on our quad xenon core machine. The result thus far was that all of the clients ran far too close to 100% cpu usage, and Varnish was serving between 140 and 150 thousand requests per second. The reason I'm telling you this is because I'm looking for input on aspects that should be tested next time I do this, which will most likely be during Christmas (far easier to borrow machine power). So far on my list, I've got: 1. Test performance over some time period when pages are evicted more frequently. (ie: X thousand pages requested repeatedly, but random expiry time). 2. Test with fewer requests per session (this is somewhat hard because the clients tend to turn into the bottleneck). 3. Test raw hot-hit cache rate (more of what I did before - get a high number). 4. Test raw performance with a huge data set that's bound to be swapped out. 5. Various tests of -sfile versus -smalloc and large datasets, combined with critbit/classic tests. 6. Find some reasonably optimal settings, then fidget around trying to find a worst-case scenario. 7. Test the effect of session lingering with really slow clients. One thing that should be noted is that the test server is limited to 1gbit, which means that for "raw req/s", we're basically forced to use tiny pages, or we just end up starving the network. The goal is to test theories regarding performance, stability and predictability. Basically find the breaking points, what's good and what's not, and what we have to care about and what we can silently ignore. As you can see, the list is getting long and this is off the top of my head, but any suggestions are welcome. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpLMdbIyku9d.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
