The next thing to do is set LogLevel to info and get the better logging from Apache error log for when the problems arise as already indicated.
Graham > On 26 Oct 2015, at 9:42 pm, Paul Royik <[email protected]> wrote: > > ServerRoot "/home/simamura/webapps/django_math/apache2" > > LoadModule alias_module modules/mod_alias.so > LoadModule authz_core_module modules/mod_authz_core.so > LoadModule dir_module modules/mod_dir.so > LoadModule env_module modules/mod_env.so > LoadModule log_config_module modules/mod_log_config.so > LoadModule mime_module modules/mod_mime.so > LoadModule rewrite_module modules/mod_rewrite.so > LoadModule setenvif_module modules/mod_setenvif.so > LoadModule wsgi_module modules/mod_wsgi.so > LoadModule unixd_module modules/mod_unixd.so > > LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" > \"%{User-Agent}i\"" combined > CustomLog /home/simamura/logs/user/access_django_math.log combined > ErrorLog /home/simamura/logs/user/error_django_math.log > > Listen 30821 > KeepAlive Off > SetEnvIf X-Forwarded-SSL on HTTPS=1 > ServerLimit 5 > StartServers 1 > MaxRequestWorkers 60 > MinSpareThreads 1 > MaxSpareThreads 3 > ThreadsPerChild 12 > > WSGIApplicationGroup %{GLOBAL} > WSGIDaemonProcess django_math processes=5 threads=12 > python-path=/home/simamura/webapps/django_math:/home/simamura/webapps/django_math/mathsite:/home/simamura/webapps/django_math/lib/python2.7 > WSGIProcessGroup django_math > WSGIRestrictEmbedded On > WSGILazyInitialization On > WSGIScriptAlias / /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py > > I changed a bit threadlimit as webfaction advised. > What should I do next? > > On Monday, October 26, 2015 at 3:13:58 AM UTC+2, Graham Dumpleton wrote: > Can you show me what your mod_wsgi configuration in Apache configuration file > actually is? Ensure that the WSGIDaemonProcess directive is included. > > Ensure that the LogLevel directive in Apache is set to ‘info’ and not ‘warn’ > and provide the actual log messages from the Apache error log from when it > occurs, including a dozen lines messages before and after so can see context. > Prior and followup messages can contain important information which you might > not think is helpful. > > Graham > >> On 23 Oct 2015, at 8:27 pm, Paul Royik <distan...@ <>gmail.com >> <http://gmail.com/>> wrote: >> >> The most long running request is 12 seconds. >> I tried to set request-timeout, but it didn't help. >> >> On Friday, October 23, 2015 at 1:25:13 AM UTC+3, Graham Dumpleton wrote: >> No MaxRequestWorkers will not necessarily help. It doesn’t control how many >> processes/threads are available in the daemon processes used by mod_wsgi. >> Even changing the processes/threads for the daemon processes is only masking >> or delaying the occurrence of the issue. You need to work out the underlying >> cause. >> >> I would suggest integrating the example code found in: >> >> >> https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces >> >> <https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces> >> >> This will give you a way of being able to get Python stack traces out of the >> application as to what it is doing when it appears to get stuck. >> >> If you have a new enough mod_wsgi version, you can also try setting on >> WSGIDaemonProcess the option: >> >> request-timeout=60 >> >> This will set a timeout on how long requests are allowed to run. If exceeded >> then the process will be forcibly restarted. >> >> This will recover from processes which are locked up and then cause who >> Apache to run out of capacity. >> >> Note that 60 means 60 seconds for a single request in case of single thread >> daemon process. >> >> Things get more complicated for multiple process and the time value relates >> to average consumed time across all active requests. So technically a single >> request can run longer than the timeout. >> >> As part of the process of restarting the process, mod_wsgi will attempt to >> dump Python stack traces as to where the code may have been running at that >> time, and so can work similar to the manual method mentioned in that link. >> >> If you play with this option, set the timeout to be twice what your longest >> request would take. Although, if you have very long running requests, what >> to set it to may be a problem in itself. >> >> Graham >> >> On 23 Oct 2015, at 4:24 am, Paul Royik <[email protected] <>> wrote: >> >> I set WSGIApplicationGroup, but it doesn't help. >> >> I used sympy all year, but problems begin only now. Maybe when I upgraded to >> Apache 2.4 >> >> Anyway, in log file I got >> [Thu Oct 22 17:08:55.697080 2015] [mpm_worker:error] [pid 658868:tid >> 140669018494784] AH00286: server reached MaxRequestWorkers setting, consider >> raising the MaxRequestWorkers setting >> [Thu Oct 22 17:09:26.399723 2015] [wsgi:error] [pid 667167:tid >> 140668926658304] [client 127.0.0.1:58424 <http://127.0.0.1:58424/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%282x%5E4%29%2F%28x%5E4-1%29&var=&a=2&b=3&steps=on >> >> <http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%282x%5E4%29%2F%28x%5E4-1%29&var=&a=2&b=3&steps=on> >> >> Will it help if I raise MaxRequestWorkers? What is the readonable number? >> >> On Thursday, October 22, 2015 at 2:13:07 PM UTC+3, Graham Dumpleton wrote: >> No you cannot change the network keep alive timeout, or not readily and it >> isn’t something you should ever do. It is quite possibly nothing to do with >> that anyway. >> >> As to WSGIApplicationGroup then it may help as I already outlined. Any >> reason you haven’t tried it as it is recommended practice to use it? >> >> What exactly are you using simpy for anyway. Looking at the documentation >> for it it does various stuff with sub processes. Heavy or complicated use of >> sub processes from inside of the context of a web application, especially >> under Apache/mod_wsgi, is generally a bad idea. This is because it may not >> play well with Apache’s process model and so something like simpy may not >> work properly or could even interfere with the process management of Apache >> or mod_wsgi. >> >> Graham >> >> On 22 Oct 2015, at 4:41 pm, Paul Royik <[email protected] <>> wrote: >> >> Problem appeared again. >> So, I will answer your questions. >> I use third-party libraries sympy and sage. Didn't have any network issues. >> Actually this happened when I sleep. >> Exact error time is 2 hours 11 minutes (this value is always the same). >> >> So, can I lower network keep alive timeout value? >> >> If I set WSGIApplicationGroup %{GLOBAL}, will this help? >> >> Thank you. >> >> On Tuesday, October 13, 2015 at 1:39:31 PM UTC+3, Graham Dumpleton wrote: >> The messages indicate that all the threads in the daemon process group were >> blocked. >> >> Initially new requests will still get added to the socket listener queue for >> the daemon processes, which has a capacity for 100 queued requests by >> default. Because though the daemon process wasn’t actually accepted them and >> removing them from the queue, on the Apache child worker side where they >> were being proxied, they would timeout. That accounts for the message: >> >> Timeout when reading response headers from daemon process >> >> When that 100 queue depth was eventually exceeded, no more requests could be >> queued up. This then resulted in attempts to connect to the daemon processes >> failing outright, which is what the error: >> >> Unable to connect to WSGI daemon process >> >> is all about. >> >> First up I would suggest you also ensure you are setting: >> >> WSGIApplicationGroup %{GLOBAL} >> >> This forces the application to run the in the main interpreter context of >> the daemon process groups. >> >> This avoids problems with third party extension modules for Python that >> haven’t been implemented correctly so as to be able to be used in sub >> interpreter contexts. These can only work in the main interpreter context. >> >> This is often a cause of thread deadlocks. >> >> That said, that you say it lasts 2 hours and then disappears is interesting. >> >> This is because 2 hours is the network keep alive timeout value. >> >> So what may be happening is request threads are contacting some backend >> service and due to some network/router issue, the connection is being >> broken. In some circumstances, depending on what software is doing with the >> connection, such a condition may not be detected until the 2 hour network >> timeout occurs. >> >> In light of that, what backend services does your application talk to and >> would that traffic traverse over any switching equipment. Have you been >> having any network issues. >> >> Graham >> >> >> On 12 Oct 2015, at 11:55 pm, Paul Royik <[email protected] <>> wrote: >> >> Hi Graham and everyone. >> >> I have a site that runs heavy math calculations using external math library. >> >> I already wrote about it nearly half a year ago. In that case there was >> timeout issue, but now I've fixed it. >> >> >> But there is another issue. >> Nearly the same time I got 503 error: Timeout when reading response headers >> from daemon process. >> This lasts for approximately 2 hours and then 503 error disappears. >> >> I use Django 1.8.4 with mod_wsgi 4.4.13. >> >> What configurations should I use in httpd.conf to avoid this error. >> How to queue requests? >> What is the number of sumultaneous requests? >> .... >> >> >> What I want to do is avoid 503 error, limit number of simultaneous requests >> and queue other requests. >> >> >> My log file: >> [Mon Oct 12 11:47:09.445618 2015] [wsgi:error] [pid 10210:tid >> 139829293606656] [client 127.0.0.1:45625 <http://127.0.0.1:45625/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py >> [Mon Oct 12 11:47:14.432179 2015] [wsgi:error] [pid 10210:tid >> 139829390120704] [client 127.0.0.1:45632 <http://127.0.0.1:45632/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/?f=tan%28x%2F2%29+%2B+%281%2F%28x%5E2%2B4%29%29&a=0&b=2&n=8&steps=on >> >> <http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/?f=tan%28x%2F2%29+%2B+%281%2F%28x%5E2%2B4%29%29&a=0&b=2&n=8&steps=on> >> [Mon Oct 12 11:47:20.201940 2015] [wsgi:error] [pid 10210:tid >> 139829400610560] [client 127.0.0.1:45700 <http://127.0.0.1:45700/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on >> >> <http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on> >> [Mon Oct 12 11:47:20.608201 2015] [wsgi:error] [pid 10210:tid >> 139829283116800] [client 127.0.0.1:45704 <http://127.0.0.1:45704/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py >> [Mon Oct 12 11:48:08.984176 2015] [wsgi:error] [pid 10210:tid >> 139829304096512] [client 127.0.0.1:45707 <http://127.0.0.1:45707/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/ >> <http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/> >> [Mon Oct 12 11:48:09.505969 2015] [wsgi:error] [pid 10210:tid >> 139829293606656] [client 127.0.0.1:45730 <http://127.0.0.1:45730/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on >> >> <http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on> >> [Mon Oct 12 11:48:14.492629 2015] [wsgi:error] [pid 10210:tid >> 139829390120704] [client 127.0.0.1:45731 <http://127.0.0.1:45731/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py >> [Mon Oct 12 11:48:20.262302 2015] [wsgi:error] [pid 10210:tid >> 139829400610560] [client 127.0.0.1:45732 <http://127.0.0.1:45732/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on >> >> <http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on> >> [Mon Oct 12 11:48:20.668421 2015] [wsgi:error] [pid 10210:tid >> 139829283116800] [client 127.0.0.1:45737 <http://127.0.0.1:45737/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> https://www.google.co.uk/ <https://www.google.co.uk/> >> [Mon Oct 12 11:49:09.044532 2015] [wsgi:error] [pid 10210:tid >> 139829304096512] [client 127.0.0.1:45744 <http://127.0.0.1:45744/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> https://www.google.co.uk/ <https://www.google.co.uk/> >> [Mon Oct 12 11:49:09.566430 2015] [wsgi:error] [pid 10210:tid >> 139829293606656] [client 127.0.0.1:45789 <http://127.0.0.1:45789/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=sin%5E5+%28x%29&var=&steps=on >> >> <http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=sin%5E5+%28x%29&var=&steps=on> >> [Mon Oct 12 11:49:14.552970 2015] [wsgi:error] [pid 10210:tid >> 139829390120704] [client 127.0.0.1:45795 <http://127.0.0.1:45795/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> https://www.google.co.uk/ <https://www.google.co.uk/> >> [Mon Oct 12 11:49:20.322179 2015] [wsgi:error] [pid 10210:tid >> 139829400610560] [client 127.0.0.1:45799 <http://127.0.0.1:45799/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/ >> <http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/> >> [Mon Oct 12 11:49:20.728784 2015] [wsgi:error] [pid 10210:tid >> 139829283116800] [client 127.0.0.1:45803 <http://127.0.0.1:45803/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on >> >> <http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on> >> [Mon Oct 12 11:50:09.104426 2015] [wsgi:error] [pid 10210:tid >> 139829304096512] [client 127.0.0.1:45816 <http://127.0.0.1:45816/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=ln+x+&var=x&steps=on >> >> <http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=ln+x+&var=x&steps=on> >> [Mon Oct 12 11:50:09.626783 2015] [wsgi:error] [pid 10210:tid >> 139829293606656] [client 127.0.0.1:45819 <http://127.0.0.1:45819/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%28y%5E%281%2F2%29%2F2%29%2B%281%2F%282*y%5E%281%2F2%29%29%29&var=&a=3&b=9 >> >> <http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%28y%5E%281%2F2%29%2F2%29%2B%281%2F%282*y%5E%281%2F2%29%29%29&var=&a=3&b=9> >> [Mon Oct 12 11:50:14.613330 2015] [wsgi:error] [pid 10210:tid >> 139829390120704] [client 127.0.0.1:45820 <http://127.0.0.1:45820/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/ >> <http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/> >> [Mon Oct 12 11:50:20.382640 2015] [wsgi:error] [pid 10210:tid >> 139829400610560] [client 127.0.0.1:45832 <http://127.0.0.1:45832/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py >> [Mon Oct 12 11:50:20.789229 2015] [wsgi:error] [pid 10210:tid >> 139829283116800] [client 127.0.0.1:45846 <http://127.0.0.1:45846/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> https://www.google.co.uk/ <https://www.google.co.uk/> >> [Mon Oct 12 11:51:09.164863 2015] [wsgi:error] [pid 10210:tid >> 139829304096512] [client 127.0.0.1:45848 <http://127.0.0.1:45848/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> https://www.google.co.uk/ <https://www.google.co.uk/> >> [Mon Oct 12 11:51:09.634602 2015] [wsgi:error] [pid 10210:tid >> 139829293606656] [client 127.0.0.1:45879 <http://127.0.0.1:45879/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&ved=0CFAQFjAJahUKEwje1pz16rzIAhXKGI4KHVrPAzA&url=http%3A%2F%2Fwww.emathhelp.net%2Fnotes%2Fcalculus-1%2Fdrawing-graphs-of-functions%2Fsteps-for-sketching-graph-of-function%2F&usg=AFQjCNGAABCyCB7rhd4yAI83Dd0NDLTrPw >> >> <http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&ved=0CFAQFjAJahUKEwje1pz16rzIAhXKGI4KHVrPAzA&url=http%3A%2F%2Fwww.emathhelp.net%2Fnotes%2Fcalculus-1%2Fdrawing-graphs-of-functions%2Fsteps-for-sketching-graph-of-function%2F&usg=AFQjCNGAABCyCB7rhd4yAI83Dd0NDLTrPw> >> [Mon Oct 12 11:51:14.639873 2015] [wsgi:error] [pid 10210:tid >> 139829390120704] [client 127.0.0.1:45914 <http://127.0.0.1:45914/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> https://www.google.co.uk/ <https://www.google.co.uk/> >> [Mon Oct 12 11:51:20.443005 2015] [wsgi:error] [pid 10210:tid >> 139829400610560] [client 127.0.0.1:45918 <http://127.0.0.1:45918/>] Timeout >> when reading response headers from daemon process 'django_math': >> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: >> https://www.google.com/ <https://www.google.com/> >> [Mon Oct 12 11:51:20.849576 2015] [wsgi:error] [pid 10210:tid >> 139829283116800] [client >> ... >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to modwsgi+u...@ <>googlegroups.com <http://googlegroups.com/>. >> To post to this group, send email to mod...@ <>googlegroups.com >> <http://googlegroups.com/>. >> Visit this group at http://groups.google.com/group/modwsgi >> <http://groups.google.com/group/modwsgi>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at http://groups.google.com/group/modwsgi > <http://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
