Hi Graham Thanks for your very prompt reply! :)
I have just investigated our most recent logfiles (in the old setup - embedded mode) - here are the figures: Requests/second: Actual: 5-15 req/sec, Expected: Up to 50 req/sec Response time: < 10ms: 50%, 10-100ms: 45%: >100ms: 5% (longest response times encountered today: 560ms) Almost entirely wsgi requests - the long running requests are all http POST to a WSGI service. This gives us a cpu-load of around 2.5 on a 2 core HyperV Ubuntu 12.04 Regarding CPU vs. I/O it will be a guess - 50/50, and the I/O is primarily network I/O to memcache (running on its own server), ldap (running on two servers in master/slave configuration) and postgresql (running on two servers in r/w resp. r configuration), meaning very little disk I/O from the python services. The server also serves PHP5 and a few static files as the application consists mainly of a number of WSGI services, some of which are using simplesamlphp for authentication purposes. We don't use mod_python. I should mention that we do have a fair amount of <Directory> configurations in the apache configuration (approx. 80) - will that have an impact on the request handling in apache? I have read your blog article regarding disabling the python interpreter in the apache parent processes - and will include that in the next attempt. I also saw one of your posts where you suggest a 1:1.5 ratio between apache threads and wsgi threads, so maybe we should aim for 150 apache threads and 100 wsgi treads (10 procs and 10 threads each)? We are looking into an upgrade of the server, but as Ubuntu 16.04 is not released yet we will try with 14.04, which includes mod_wsgi 4.3.0. Would that be recent enough, or would it be wiser to go with the latest release (4.4.21)? Do you know which version will go into 16.04? Lars Den torsdag den 3. marts 2016 kl. 23.59.29 UTC+1 skrev Graham Dumpleton: > > What number of requests/sec does your web site need to handle (as opposed > to what you are getting)? > > What is the average response times for requests? > > What long running requests do you get? How long and how often? > > Is the code your request handlers run primarily CPU or I/O bound? > > What other Apache modules for other languages are you loading? Eg., > mod_python, mod_php etc. > > My initial impression is that you are creating way more daemon processes > than you need. Creating too many can actually make things worse and be > detrimental to memory usage and performance. > > The resident size of the Apache child worker processes suggests you aren't > disabling the creation of the Python interpreter in the Apache child > processes. > > http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html > > You are also using a very old mod_wsgi version. More recent versions > better control memory usage to work around some oddities around how Apache > handles memory and buffering. It is not recommended to be using such an > older mod_wsgi version. It isn’t worth digging to much further into this > unless you are on recent version. > > Can you possibly upgrade to newer mod_wsgi? > > Graham > > On 3 Mar 2016, at 11:32 PM, Lars Westermann <[email protected] > <javascript:>> wrote: > > I have a webserver running apache (2.4.10) and mod_wsgi (3.4) > > I have seen a cpu-load around 2.5 (on a 2 core HyperV-2012 machine) - and > cpu-utilization well below 10%. > After searching for information on how to improve things, I changed from > running mod_wsgi in embedded mode to > running it in daemon mode - in the hope that offloading the python code > from apache would help. > > Unfortunately this is not the case. Cpu-load increased to around 8, > cpu-utilization was well over 10% - so clearly something is wrong! > > I am using apache in prefork mode (yes, I know, but php5 module is not > threadsafe...) - with this configuration: > > <IfModule mpm_prefork_module> > StartServers 5 > MinSpareServers 5 > MaxSpareServers 10 > MaxRequestWorkers 150 > MaxConnectionsPerChild 0 > </IfModule> > > At the global configuration level I have this: > > WSGIDaemonProcess web-deploy processes=30 threads=5 > display-name=wd-wsgi-daemon > WSGIProcessGroup web-deploy > > > Process tree (using htop) shows this: > > 1 [|||||||||| > 12.8%] Tasks: 81, 289 thr; 1 running > 2 [|||||||| > 9.3%] Load average: 6.25 6.54 5.99 > > Mem[||||||||||||||||||||||||||||||||||||||||||||||||||||||5110/9996MB] > Uptime: 72 days, 03:19:58 > Swp[| 10/1021MB] > > PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command > 1 root 20 0 90576 3288 2468 S 0.0 0.0 0:04.96 /sbin/init > 46975 root 20 0 424M 20424 12792 S 0.0 0.2 0:02.06 ├─ > /usr/sbin/apache2 -k start > 51598 www-data 20 0 429M 24048 12744 S 0.0 0.2 0:12.50 │ ├─ > /usr/sbin/apache2 -k start > 51561 www-data 20 0 429M 24132 12744 S 0.0 0.2 0:12.29 │ ├─ > /usr/sbin/apache2 -k start > 47453 www-data 20 0 432M 27444 13568 S 0.0 0.3 0:40.11 │ ├─ > /usr/sbin/apache2 -k start > 47006 www-data 20 0 1414M 164M 7772 S 0.0 1.7 1:20.43 │ ├─ > wd-wsgi-daemon -k start > 47214 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:14.50 │ │ ├─ > wd-wsgi-daemon -k start > 47205 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:13.08 │ │ ├─ > wd-wsgi-daemon -k start > 47204 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:12.73 │ │ ├─ > wd-wsgi-daemon -k start > 47203 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:07.11 │ │ ├─ > wd-wsgi-daemon -k start > 47202 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:11.95 │ │ ├─ > wd-wsgi-daemon -k start > 47201 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:01.48 │ │ ├─ > wd-wsgi-daemon -k start > 47200 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:00.00 │ │ ├─ > wd-wsgi-daemon -k start > 26045 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:00.08 │ │ ├─ > wd-wsgi-daemon -k start > 26043 www-data 20 0 1414M 164M 7772 S 0.0 1.7 0:00.08 │ │ └─ > wd-wsgi-daemon -k start > 47005 www-data 20 0 1478M 147M 7108 S 0.0 1.5 1:19.79 │ ├─ > wd-wsgi-daemon -k start > 47212 www-data 20 0 1478M 147M 7108 S 0.0 1.5 0:17.69 │ │ ├─ > wd-wsgi-daemon -k start > 47211 www-data 20 0 1478M 147M 7108 S 0.0 1.5 0:09.00 │ │ ├─ > wd-wsgi-daemon -k start > > Strangely there are 9 threads in each wsgi-daemon (I would expect 8 > including the 3 used for process control), > ref: > http://blog.dscpl.com.au/2014/02/use-of-threading-in-modwsgi-daemon-mode.html > > WSGI daemons use quite a lot of memory - and I think we have too many > wsgi-daemon processes. The reason for the many processes is that > many of the python services use other network services (PostgreSQL, LDAP, > memcache), and I have read that the threads share a number > of common resources in this respect. > > When using top, it looks like apache processes are using the > cpu-resources, wsgi-daemon processes don't even show up in the top of the > list. > > Apache handles approx. 75 requests/second with the above shown cpu-load. > > After reverting to the previous configuration (no wsgi-daemons), htop > shows this: > > 1 [|||||| > 8.6%] Tasks: 51, 36 thr; 1 running > 2 [| > 0.7%] Load average: 1.87 2.20 3.28 > Mem[||||||||||||||||||||||||||||||||||||||||||| > 1917/9996MB] Uptime: 72 days, 03:59:56 > Swp[| 11/1021MB] > > PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command > 39518 www-data 20 0 1008M 126M 14552 S 1.0 1.3 0:25.87 > /usr/sbin/apache2 -k start > 39405 www-data 20 0 1080M 129M 15280 S 1.0 1.3 0:24.94 > /usr/sbin/apache2 -k start > 39517 www-data 20 0 1068M 131M 14392 S 1.0 1.3 0:24.60 > /usr/sbin/apache2 -k start > 39560 www-data 20 0 1139M 131M 15116 S 1.0 1.3 0:26.76 > /usr/sbin/apache2 -k start > 39470 root 20 0 94320 4884 2812 R 0.0 0.0 0:14.37 htop > 39514 www-data 20 0 1150M 141M 14392 S 0.0 1.4 0:21.83 > /usr/sbin/apache2 -k start > 39387 www-data 20 0 1032M 147M 17844 S 0.0 1.5 0:26.35 > /usr/sbin/apache2 -k start > 39468 www-data 20 0 1145M 135M 14400 S 0.0 1.4 0:25.96 > /usr/sbin/apache2 -k start > 39467 www-data 20 0 1019M 147M 16700 S 0.0 1.5 0:24.94 > /usr/sbin/apache2 -k start > 39466 www-data 20 0 1145M 144M 15688 S 0.0 1.5 0:24.90 > /usr/sbin/apache2 -k start > > > Does anyone have a clue, or just a hint to what to do next? > > Best Regards, > Lars > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] <javascript:> > . > Visit this group at https://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
