It isn't out of the ordinary for response times to go up as number of concurrent requests is increased. It is the sum total of the interactive affects of multiple processes, multiple threads and the Python GIL.
What you aren't graphing is throughput, so relying on response times alone is deceiving. Take for example the charts in my post at: https://plus.google.com/114657481176404420131/posts/G1jM6WW3Pnu http://dl.dropbox.com/u/22571016/SafariScreenSnapz107.gif http://dl.dropbox.com/u/22571016/SafariScreenSnapz108.gif The response times grow out to 150ms for a WSGI hello world in the worst of the two configurations, but that is quite reasonable when you consider that though is scaling up to a throughput of 3500 requests per minute at high levels of concurrency. For the better configuration of the two, response times grow out only to 100ms, but throughput goes up to 4500 requests per minute. Whether one configuration is better than another will depend on your specific application and whether they are cpu intensive tasks for i/o bound tasks. So individual response times alone are not a good measure, you also need to look at capacity based on throughput. As throughput increases, you work out whether the decrease in response time is acceptable. At that point for your application you look at changing processes vs threads at server level to make it work more efficiently, but usually more importantly tuning your application to cut down application and database bottlenecks. Get rid of the worst and you will bring down response times that way as well. Long running requests are an issue and that is where you need to look at total number of threads across all processes in both Apache MPM setting and daemon mode settings. In those graphs you will see a few things which exploring to help tune those things. These are thread utilisation and queuing time. Thread utilisation is showing how much of the capacity of the available threads is being utilised and so when that itself is becoming a restriction. The queuing time is how long requests are sitting there doing nothing between when Apache first accepts requests by MPM controlled worker process and when daemon mode process then picks it up and handles it. So, it is showing how much requests are getting backlogged, with that queuing time being on top of response time as measure by the application. Again an indication of two few processes/threads for daemon mode for that traffic profile. If you are happy to try mod_wsgi 4.0, you can get access to both queuing time and also thread utilisation. It will all change at some point, but the current New Relic Python agent is able to grab the data and you can graph it with custom views to get those charts I link to. As to your New Relic account, I can't see to find it by searching for email or name. You might let me know the account number that shows in URL so can look at what data you are getting and if you try mod_wsgi 4.0, can give you the custom view definition you can setup to get that chart. BTW, don't entirely focus on the application side either. Although you may get your application performance times down to 100ms levels, the users will not care much if they are still seeing 6 second page load times because of page rendering times. Better user satisfaction can therefore often be more quickly by improving the HTML/JavaScript sent back to the browser. Graham On 20 January 2012 06:19, Daniel Benamy <[email protected]> wrote: > On Wed, Jan 18, 2012 at 7:11 PM, Graham Dumpleton > <[email protected]> wrote: >> Sorry about the delay in responding, Google groups never sent me a >> moderation email or Google is putting its own emails in the spam folder. I >> am also traveling away from home right now which hasn't help as had limited >> Internet access. > > No worries. I appreciate the help! > >> Can you you post some details about what Apache MPM you are using, what the >> MPM configuration is, plus whether using mod_wsgi daemon mode and if so what >> configuration for that. > > The exact setup I used for the benchmarks is below the data and graph > at > https://docs.google.com/spreadsheet/pub?hl=en_US&hl=en_US&key=0AurdDQB5QBe7dGFncnlUNkdKMVJ4NnYtRjhjaGFIeFE&output=html. > > To answer your question, it's mpm-worker and mod_wsgi daemon mode. > StartServers 2 > MinSpareThreads 25 > MaxSpareThreads 75 > ThreadLimit 64 > ThreadsPerChild 25 > MaxClients 150 > MaxRequestsPerChild 0 > WSGIDaemonProcess example.com processes=2 threads=15 display-name=%{GROUP} > > On our actual site, some settings are different but we are using > mpm-worker and mod_wsgi daemon mode. > >> As far as getting much deeper insight into where time is being consumed, >> suggest start reading about what New Relic is. I work there these days and > > I've heard great things about it and have actually already started the > trial period. It's super slick! If you have any thoughts about which > reports to use, I'd love to hear them. It's likely that we'll keep it > on at least one server after the trial period. > > Thanks again, > Dan > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
