Can you show me what your mod_wsgi configuration in Apache configuration file actually is? Ensure that the WSGIDaemonProcess directive is included.
Ensure that the LogLevel directive in Apache is set to ‘info’ and not ‘warn’ and provide the actual log messages from the Apache error log from when it occurs, including a dozen lines messages before and after so can see context. Prior and followup messages can contain important information which you might not think is helpful. Graham > On 23 Oct 2015, at 8:27 pm, Paul Royik <[email protected]> wrote: > > The most long running request is 12 seconds. > I tried to set request-timeout, but it didn't help. > > On Friday, October 23, 2015 at 1:25:13 AM UTC+3, Graham Dumpleton wrote: > No MaxRequestWorkers will not necessarily help. It doesn’t control how many > processes/threads are available in the daemon processes used by mod_wsgi. > Even changing the processes/threads for the daemon processes is only masking > or delaying the occurrence of the issue. You need to work out the underlying > cause. > > I would suggest integrating the example code found in: > > > https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces > > <https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces> > > This will give you a way of being able to get Python stack traces out of the > application as to what it is doing when it appears to get stuck. > > If you have a new enough mod_wsgi version, you can also try setting on > WSGIDaemonProcess the option: > > request-timeout=60 > > This will set a timeout on how long requests are allowed to run. If exceeded > then the process will be forcibly restarted. > > This will recover from processes which are locked up and then cause who > Apache to run out of capacity. > > Note that 60 means 60 seconds for a single request in case of single thread > daemon process. > > Things get more complicated for multiple process and the time value relates > to average consumed time across all active requests. So technically a single > request can run longer than the timeout. > > As part of the process of restarting the process, mod_wsgi will attempt to > dump Python stack traces as to where the code may have been running at that > time, and so can work similar to the manual method mentioned in that link. > > If you play with this option, set the timeout to be twice what your longest > request would take. Although, if you have very long running requests, what to > set it to may be a problem in itself. > > Graham > > On 23 Oct 2015, at 4:24 am, Paul Royik <[email protected] <javascript:>> > wrote: > > I set WSGIApplicationGroup, but it doesn't help. > > I used sympy all year, but problems begin only now. Maybe when I upgraded to > Apache 2.4 > > Anyway, in log file I got > [Thu Oct 22 17:08:55.697080 2015] [mpm_worker:error] [pid 658868:tid > 140669018494784] AH00286: server reached MaxRequestWorkers setting, consider > raising the MaxRequestWorkers setting > [Thu Oct 22 17:09:26.399723 2015] [wsgi:error] [pid 667167:tid > 140668926658304] [client 127.0.0.1:58424 <http://127.0.0.1:58424/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%282x%5E4%29%2F%28x%5E4-1%29&var=&a=2&b=3&steps=on > > <http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%282x%5E4%29%2F%28x%5E4-1%29&var=&a=2&b=3&steps=on> > > Will it help if I raise MaxRequestWorkers? What is the readonable number? > > On Thursday, October 22, 2015 at 2:13:07 PM UTC+3, Graham Dumpleton wrote: > No you cannot change the network keep alive timeout, or not readily and it > isn’t something you should ever do. It is quite possibly nothing to do with > that anyway. > > As to WSGIApplicationGroup then it may help as I already outlined. Any reason > you haven’t tried it as it is recommended practice to use it? > > What exactly are you using simpy for anyway. Looking at the documentation for > it it does various stuff with sub processes. Heavy or complicated use of sub > processes from inside of the context of a web application, especially under > Apache/mod_wsgi, is generally a bad idea. This is because it may not play > well with Apache’s process model and so something like simpy may not work > properly or could even interfere with the process management of Apache or > mod_wsgi. > > Graham > > On 22 Oct 2015, at 4:41 pm, Paul Royik <[email protected] <>> wrote: > > Problem appeared again. > So, I will answer your questions. > I use third-party libraries sympy and sage. Didn't have any network issues. > Actually this happened when I sleep. > Exact error time is 2 hours 11 minutes (this value is always the same). > > So, can I lower network keep alive timeout value? > > If I set WSGIApplicationGroup %{GLOBAL}, will this help? > > Thank you. > > On Tuesday, October 13, 2015 at 1:39:31 PM UTC+3, Graham Dumpleton wrote: > The messages indicate that all the threads in the daemon process group were > blocked. > > Initially new requests will still get added to the socket listener queue for > the daemon processes, which has a capacity for 100 queued requests by > default. Because though the daemon process wasn’t actually accepted them and > removing them from the queue, on the Apache child worker side where they were > being proxied, they would timeout. That accounts for the message: > > Timeout when reading response headers from daemon process > > When that 100 queue depth was eventually exceeded, no more requests could be > queued up. This then resulted in attempts to connect to the daemon processes > failing outright, which is what the error: > > Unable to connect to WSGI daemon process > > is all about. > > First up I would suggest you also ensure you are setting: > > WSGIApplicationGroup %{GLOBAL} > > This forces the application to run the in the main interpreter context of the > daemon process groups. > > This avoids problems with third party extension modules for Python that > haven’t been implemented correctly so as to be able to be used in sub > interpreter contexts. These can only work in the main interpreter context. > > This is often a cause of thread deadlocks. > > That said, that you say it lasts 2 hours and then disappears is interesting. > > This is because 2 hours is the network keep alive timeout value. > > So what may be happening is request threads are contacting some backend > service and due to some network/router issue, the connection is being broken. > In some circumstances, depending on what software is doing with the > connection, such a condition may not be detected until the 2 hour network > timeout occurs. > > In light of that, what backend services does your application talk to and > would that traffic traverse over any switching equipment. Have you been > having any network issues. > > Graham > > > On 12 Oct 2015, at 11:55 pm, Paul Royik <[email protected] <>> wrote: > > Hi Graham and everyone. > > I have a site that runs heavy math calculations using external math library. > > I already wrote about it nearly half a year ago. In that case there was > timeout issue, but now I've fixed it. > > > But there is another issue. > Nearly the same time I got 503 error: Timeout when reading response headers > from daemon process. > This lasts for approximately 2 hours and then 503 error disappears. > > I use Django 1.8.4 with mod_wsgi 4.4.13. > > What configurations should I use in httpd.conf to avoid this error. > How to queue requests? > What is the number of sumultaneous requests? > .... > > > What I want to do is avoid 503 error, limit number of simultaneous requests > and queue other requests. > > > My log file: > [Mon Oct 12 11:47:09.445618 2015] [wsgi:error] [pid 10210:tid > 139829293606656] [client 127.0.0.1:45625 <http://127.0.0.1:45625/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py > [Mon Oct 12 11:47:14.432179 2015] [wsgi:error] [pid 10210:tid > 139829390120704] [client 127.0.0.1:45632 <http://127.0.0.1:45632/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/?f=tan%28x%2F2%29+%2B+%281%2F%28x%5E2%2B4%29%29&a=0&b=2&n=8&steps=on > > <http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/?f=tan%28x%2F2%29+%2B+%281%2F%28x%5E2%2B4%29%29&a=0&b=2&n=8&steps=on> > [Mon Oct 12 11:47:20.201940 2015] [wsgi:error] [pid 10210:tid > 139829400610560] [client 127.0.0.1:45700 <http://127.0.0.1:45700/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on > > <http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on> > [Mon Oct 12 11:47:20.608201 2015] [wsgi:error] [pid 10210:tid > 139829283116800] [client 127.0.0.1:45704 <http://127.0.0.1:45704/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py > [Mon Oct 12 11:48:08.984176 2015] [wsgi:error] [pid 10210:tid > 139829304096512] [client 127.0.0.1:45707 <http://127.0.0.1:45707/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/ > <http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/> > [Mon Oct 12 11:48:09.505969 2015] [wsgi:error] [pid 10210:tid > 139829293606656] [client 127.0.0.1:45730 <http://127.0.0.1:45730/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on > > <http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on> > [Mon Oct 12 11:48:14.492629 2015] [wsgi:error] [pid 10210:tid > 139829390120704] [client 127.0.0.1:45731 <http://127.0.0.1:45731/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py > [Mon Oct 12 11:48:20.262302 2015] [wsgi:error] [pid 10210:tid > 139829400610560] [client 127.0.0.1:45732 <http://127.0.0.1:45732/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on > > <http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on> > [Mon Oct 12 11:48:20.668421 2015] [wsgi:error] [pid 10210:tid > 139829283116800] [client 127.0.0.1:45737 <http://127.0.0.1:45737/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > https://www.google.co.uk/ <https://www.google.co.uk/> > [Mon Oct 12 11:49:09.044532 2015] [wsgi:error] [pid 10210:tid > 139829304096512] [client 127.0.0.1:45744 <http://127.0.0.1:45744/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > https://www.google.co.uk/ <https://www.google.co.uk/> > [Mon Oct 12 11:49:09.566430 2015] [wsgi:error] [pid 10210:tid > 139829293606656] [client 127.0.0.1:45789 <http://127.0.0.1:45789/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=sin%5E5+%28x%29&var=&steps=on > > <http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=sin%5E5+%28x%29&var=&steps=on> > [Mon Oct 12 11:49:14.552970 2015] [wsgi:error] [pid 10210:tid > 139829390120704] [client 127.0.0.1:45795 <http://127.0.0.1:45795/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > https://www.google.co.uk/ <https://www.google.co.uk/> > [Mon Oct 12 11:49:20.322179 2015] [wsgi:error] [pid 10210:tid > 139829400610560] [client 127.0.0.1:45799 <http://127.0.0.1:45799/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/ > <http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/> > [Mon Oct 12 11:49:20.728784 2015] [wsgi:error] [pid 10210:tid > 139829283116800] [client 127.0.0.1:45803 <http://127.0.0.1:45803/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on > > <http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on> > [Mon Oct 12 11:50:09.104426 2015] [wsgi:error] [pid 10210:tid > 139829304096512] [client 127.0.0.1:45816 <http://127.0.0.1:45816/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=ln+x+&var=x&steps=on > > <http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=ln+x+&var=x&steps=on> > [Mon Oct 12 11:50:09.626783 2015] [wsgi:error] [pid 10210:tid > 139829293606656] [client 127.0.0.1:45819 <http://127.0.0.1:45819/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%28y%5E%281%2F2%29%2F2%29%2B%281%2F%282*y%5E%281%2F2%29%29%29&var=&a=3&b=9 > > <http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%28y%5E%281%2F2%29%2F2%29%2B%281%2F%282*y%5E%281%2F2%29%29%29&var=&a=3&b=9> > [Mon Oct 12 11:50:14.613330 2015] [wsgi:error] [pid 10210:tid > 139829390120704] [client 127.0.0.1:45820 <http://127.0.0.1:45820/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/ > <http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/> > [Mon Oct 12 11:50:20.382640 2015] [wsgi:error] [pid 10210:tid > 139829400610560] [client 127.0.0.1:45832 <http://127.0.0.1:45832/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py > [Mon Oct 12 11:50:20.789229 2015] [wsgi:error] [pid 10210:tid > 139829283116800] [client 127.0.0.1:45846 <http://127.0.0.1:45846/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > https://www.google.co.uk/ <https://www.google.co.uk/> > [Mon Oct 12 11:51:09.164863 2015] [wsgi:error] [pid 10210:tid > 139829304096512] [client 127.0.0.1:45848 <http://127.0.0.1:45848/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > https://www.google.co.uk/ <https://www.google.co.uk/> > [Mon Oct 12 11:51:09.634602 2015] [wsgi:error] [pid 10210:tid > 139829293606656] [client 127.0.0.1:45879 <http://127.0.0.1:45879/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&ved=0CFAQFjAJahUKEwje1pz16rzIAhXKGI4KHVrPAzA&url=http%3A%2F%2Fwww.emathhelp.net%2Fnotes%2Fcalculus-1%2Fdrawing-graphs-of-functions%2Fsteps-for-sketching-graph-of-function%2F&usg=AFQjCNGAABCyCB7rhd4yAI83Dd0NDLTrPw > > <http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&ved=0CFAQFjAJahUKEwje1pz16rzIAhXKGI4KHVrPAzA&url=http%3A%2F%2Fwww.emathhelp.net%2Fnotes%2Fcalculus-1%2Fdrawing-graphs-of-functions%2Fsteps-for-sketching-graph-of-function%2F&usg=AFQjCNGAABCyCB7rhd4yAI83Dd0NDLTrPw> > [Mon Oct 12 11:51:14.639873 2015] [wsgi:error] [pid 10210:tid > 139829390120704] [client 127.0.0.1:45914 <http://127.0.0.1:45914/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > https://www.google.co.uk/ <https://www.google.co.uk/> > [Mon Oct 12 11:51:20.443005 2015] [wsgi:error] [pid 10210:tid > 139829400610560] [client 127.0.0.1:45918 <http://127.0.0.1:45918/>] Timeout > when reading response headers from daemon process 'django_math': > /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer: > https://www.google.com/ <https://www.google.com/> > [Mon Oct 12 11:51:20.849576 2015] [wsgi:error] [pid 10210:tid > 139829283116800] [client > ... > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at http://groups.google.com/group/modwsgi > <http://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
