ServerRoot "/home/simamura/webapps/django_math/apache2"
LoadModule alias_module modules/mod_alias.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule wsgi_module modules/mod_wsgi.so
LoadModule unixd_module modules/mod_unixd.so
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\"" combined
CustomLog /home/simamura/logs/user/access_django_math.log combined
ErrorLog /home/simamura/logs/user/error_django_math.log
Listen 30821
KeepAlive Off
SetEnvIf X-Forwarded-SSL on HTTPS=1
ServerLimit 5
StartServers 1
MaxRequestWorkers 60
MinSpareThreads 1
MaxSpareThreads 3
ThreadsPerChild 12
WSGIApplicationGroup %{GLOBAL}
WSGIDaemonProcess django_math processes=5 threads=12
python-path=/home/simamura/webapps/django_math:/home/simamura/webapps/django_math/mathsite:/home/simamura/webapps/django_math/lib/python2.7
WSGIProcessGroup django_math
WSGIRestrictEmbedded On
WSGILazyInitialization On
WSGIScriptAlias /
/home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
I changed a bit threadlimit as webfaction advised.
What should I do next?
On Monday, October 26, 2015 at 3:13:58 AM UTC+2, Graham Dumpleton wrote:
>
> Can you show me what your mod_wsgi configuration in Apache configuration
> file actually is? Ensure that the WSGIDaemonProcess directive is included.
>
> Ensure that the LogLevel directive in Apache is set to ‘info’ and not
> ‘warn’ and provide the actual log messages from the Apache error log from
> when it occurs, including a dozen lines messages before and after so can
> see context. Prior and followup messages can contain important information
> which you might not think is helpful.
>
> Graham
>
> On 23 Oct 2015, at 8:27 pm, Paul Royik <[email protected] <javascript:>>
> wrote:
>
> The most long running request is 12 seconds.
> I tried to set request-timeout, but it didn't help.
>
> On Friday, October 23, 2015 at 1:25:13 AM UTC+3, Graham Dumpleton wrote:
>>
>> No MaxRequestWorkers will not necessarily help. It doesn’t control how
>> many processes/threads are available in the daemon processes used by
>> mod_wsgi. Even changing the processes/threads for the daemon processes is
>> only masking or delaying the occurrence of the issue. You need to work out
>> the underlying cause.
>>
>> I would suggest integrating the example code found in:
>>
>>
>> https://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces
>>
>> This will give you a way of being able to get Python stack traces out of
>> the application as to what it is doing when it appears to get stuck.
>>
>> If you have a new enough mod_wsgi version, you can also try setting on
>> WSGIDaemonProcess the option:
>>
>> request-timeout=60
>>
>> This will set a timeout on how long requests are allowed to run. If
>> exceeded then the process will be forcibly restarted.
>>
>> This will recover from processes which are locked up and then cause who
>> Apache to run out of capacity.
>>
>> Note that 60 means 60 seconds for a single request in case of single
>> thread daemon process.
>>
>> Things get more complicated for multiple process and the time value
>> relates to average consumed time across all active requests. So technically
>> a single request can run longer than the timeout.
>>
>> As part of the process of restarting the process, mod_wsgi will attempt
>> to dump Python stack traces as to where the code may have been running at
>> that time, and so can work similar to the manual method mentioned in that
>> link.
>>
>> If you play with this option, set the timeout to be twice what your
>> longest request would take. Although, if you have very long running
>> requests, what to set it to may be a problem in itself.
>>
>> Graham
>>
>> On 23 Oct 2015, at 4:24 am, Paul Royik <[email protected]> wrote:
>>
>> I set WSGIApplicationGroup, but it doesn't help.
>>
>> I used sympy all year, but problems begin only now. Maybe when I upgraded
>> to Apache 2.4
>>
>> Anyway, in log file I got
>> [Thu Oct 22 17:08:55.697080 2015] [mpm_worker:error] [pid 658868:tid
>> 140669018494784] AH00286: server reached MaxRequestWorkers setting,
>> consider raising the MaxRequestWorkers setting
>> [Thu Oct 22 17:09:26.399723 2015] [wsgi:error] [pid 667167:tid
>> 140668926658304] [client 127.0.0.1:58424] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%282x%5E4%29%2F%28x%5E4-1%29&var=&a=2&b=3&steps=on
>>
>> Will it help if I raise MaxRequestWorkers? What is the readonable number?
>>
>> On Thursday, October 22, 2015 at 2:13:07 PM UTC+3, Graham Dumpleton wrote:
>>
>> No you cannot change the network keep alive timeout, or not readily and
>> it isn’t something you should ever do. It is quite possibly nothing to do
>> with that anyway.
>>
>> As to WSGIApplicationGroup then it may help as I already outlined. Any
>> reason you haven’t tried it as it is recommended practice to use it?
>>
>> What exactly are you using simpy for anyway. Looking at the documentation
>> for it it does various stuff with sub processes. Heavy or complicated use
>> of sub processes from inside of the context of a web application,
>> especially under Apache/mod_wsgi, is generally a bad idea. This is because
>> it may not play well with Apache’s process model and so something like
>> simpy may not work properly or could even interfere with the process
>> management of Apache or mod_wsgi.
>>
>> Graham
>>
>> On 22 Oct 2015, at 4:41 pm, Paul Royik <[email protected]> wrote:
>>
>> Problem appeared again.
>> So, I will answer your questions.
>> I use third-party libraries sympy and sage. Didn't have any network
>> issues.
>> Actually this happened when I sleep.
>> Exact error time is 2 hours 11 minutes (this value is always the same).
>>
>> So, can I lower network keep alive timeout value?
>>
>> If I set WSGIApplicationGroup %{GLOBAL}, will this help?
>>
>> Thank you.
>>
>> On Tuesday, October 13, 2015 at 1:39:31 PM UTC+3, Graham Dumpleton wrote:
>>
>> The messages indicate that all the threads in the daemon process group
>> were blocked.
>>
>> Initially new requests will still get added to the socket listener queue
>> for the daemon processes, which has a capacity for 100 queued requests by
>> default. Because though the daemon process wasn’t actually accepted them
>> and removing them from the queue, on the Apache child worker side where
>> they were being proxied, they would timeout. That accounts for the message:
>>
>> Timeout when reading response headers from daemon process
>>
>> When that 100 queue depth was eventually exceeded, no more requests could
>> be queued up. This then resulted in attempts to connect to the daemon
>> processes failing outright, which is what the error:
>>
>> Unable to connect to WSGI daemon process
>>
>> is all about.
>>
>> First up I would suggest you also ensure you are setting:
>>
>> WSGIApplicationGroup %{GLOBAL}
>>
>> This forces the application to run the in the main interpreter context of
>> the daemon process groups.
>>
>> This avoids problems with third party extension modules for Python that
>> haven’t been implemented correctly so as to be able to be used in sub
>> interpreter contexts. These can only work in the main interpreter context.
>>
>> This is often a cause of thread deadlocks.
>>
>> That said, that you say it lasts 2 hours and then disappears is
>> interesting.
>>
>> This is because 2 hours is the network keep alive timeout value.
>>
>> So what may be happening is request threads are contacting some backend
>> service and due to some network/router issue, the connection is being
>> broken. In some circumstances, depending on what software is doing with the
>> connection, such a condition may not be detected until the 2 hour network
>> timeout occurs.
>>
>> In light of that, what backend services does your application talk to and
>> would that traffic traverse over any switching equipment. Have you been
>> having any network issues.
>>
>> Graham
>>
>>
>> On 12 Oct 2015, at 11:55 pm, Paul Royik <[email protected]> wrote:
>>
>> Hi Graham and everyone.
>>
>> I have a site that runs heavy math calculations using external math
>> library.
>>
>> I already wrote about it nearly half a year ago. In that case there was
>> timeout issue, but now I've fixed it.
>>
>>
>> But there is another issue.
>> Nearly the same time I got 503 error: Timeout when reading response
>> headers from daemon process.
>> This lasts for approximately 2 hours and then 503 error disappears.
>>
>> I use Django 1.8.4 with mod_wsgi 4.4.13.
>>
>> What configurations should I use in httpd.conf to avoid this error.
>> How to queue requests?
>> What is the number of sumultaneous requests?
>> ....
>>
>>
>> What I want to do is avoid 503 error, limit number of simultaneous
>> requests and queue other requests.
>>
>>
>> My log file:
>> [Mon Oct 12 11:47:09.445618 2015] [wsgi:error] [pid 10210:tid
>> 139829293606656] [client 127.0.0.1:45625] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
>> [Mon Oct 12 11:47:14.432179 2015] [wsgi:error] [pid 10210:tid
>> 139829390120704] [client 127.0.0.1:45632] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/?f=tan%28x%2F2%29+%2B+%281%2F%28x%5E2%2B4%29%29&a=0&b=2&n=8&steps=on
>> [Mon Oct 12 11:47:20.201940 2015] [wsgi:error] [pid 10210:tid
>> 139829400610560] [client 127.0.0.1:45700] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on
>> [Mon Oct 12 11:47:20.608201 2015] [wsgi:error] [pid 10210:tid
>> 139829283116800] [client 127.0.0.1:45704] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
>> [Mon Oct 12 11:48:08.984176 2015] [wsgi:error] [pid 10210:tid
>> 139829304096512] [client 127.0.0.1:45707] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/
>> [Mon Oct 12 11:48:09.505969 2015] [wsgi:error] [pid 10210:tid
>> 139829293606656] [client 127.0.0.1:45730] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on
>> [Mon Oct 12 11:48:14.492629 2015] [wsgi:error] [pid 10210:tid
>> 139829390120704] [client 127.0.0.1:45731] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
>> [Mon Oct 12 11:48:20.262302 2015] [wsgi:error] [pid 10210:tid
>> 139829400610560] [client 127.0.0.1:45732] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/algebra-1/synthetic-division-calculator/?numer=2x%5E3-7x%5E2%2B12x-9&denom=2x-3&steps=on
>> [Mon Oct 12 11:48:20.668421 2015] [wsgi:error] [pid 10210:tid
>> 139829283116800] [client 127.0.0.1:45737] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> https://www.google.co.uk/
>> [Mon Oct 12 11:49:09.044532 2015] [wsgi:error] [pid 10210:tid
>> 139829304096512] [client 127.0.0.1:45744] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> https://www.google.co.uk/
>> [Mon Oct 12 11:49:09.566430 2015] [wsgi:error] [pid 10210:tid
>> 139829293606656] [client 127.0.0.1:45789] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=sin%5E5+%28x%29&var=&steps=on
>> [Mon Oct 12 11:49:14.552970 2015] [wsgi:error] [pid 10210:tid
>> 139829390120704] [client 127.0.0.1:45795] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> https://www.google.co.uk/
>> [Mon Oct 12 11:49:20.322179 2015] [wsgi:error] [pid 10210:tid
>> 139829400610560] [client 127.0.0.1:45799] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/simpsons-rule-calculator/
>> [Mon Oct 12 11:49:20.728784 2015] [wsgi:error] [pid 10210:tid
>> 139829283116800] [client 127.0.0.1:45803] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/algebra-2/partial-fraction-decomposition-calculator/?numer=4x%5E&denom=x%5E2%28x%5E2%2B1%29&steps=on
>> [Mon Oct 12 11:50:09.104426 2015] [wsgi:error] [pid 10210:tid
>> 139829304096512] [client 127.0.0.1:45816] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/integral-calculator/?f=ln+x+&var=x&steps=on
>> [Mon Oct 12 11:50:09.626783 2015] [wsgi:error] [pid 10210:tid
>> 139829293606656] [client 127.0.0.1:45819] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/definite-integral-calculator/?f=%28y%5E%281%2F2%29%2F2%29%2B%281%2F%282*y%5E%281%2F2%29%29%29&var=&a=3&b=9
>> [Mon Oct 12 11:50:14.613330 2015] [wsgi:error] [pid 10210:tid
>> 139829390120704] [client 127.0.0.1:45820] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.emathhelp.net/calculators/calculus-2/trapezoidal-rule-calculator/
>> [Mon Oct 12 11:50:20.382640 2015] [wsgi:error] [pid 10210:tid
>> 139829400610560] [client 127.0.0.1:45832] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py
>> [Mon Oct 12 11:50:20.789229 2015] [wsgi:error] [pid 10210:tid
>> 139829283116800] [client 127.0.0.1:45846] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> https://www.google.co.uk/
>> [Mon Oct 12 11:51:09.164863 2015] [wsgi:error] [pid 10210:tid
>> 139829304096512] [client 127.0.0.1:45848] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> https://www.google.co.uk/
>> [Mon Oct 12 11:51:09.634602 2015] [wsgi:error] [pid 10210:tid
>> 139829293606656] [client 127.0.0.1:45879] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> http://www.google.co.in/url?sa=t&rct=j&q=&esrc=s&source=web&cd=10&ved=0CFAQFjAJahUKEwje1pz16rzIAhXKGI4KHVrPAzA&url=http%3A%2F%2Fwww.emathhelp.net%2Fnotes%2Fcalculus-1%2Fdrawing-graphs-of-functions%2Fsteps-for-sketching-graph-of-function%2F&usg=AFQjCNGAABCyCB7rhd4yAI83Dd0NDLTrPw
>> [Mon Oct 12 11:51:14.639873 2015] [wsgi:error] [pid 10210:tid
>> 139829390120704] [client 127.0.0.1:45914] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> https://www.google.co.uk/
>> [Mon Oct 12 11:51:20.443005 2015] [wsgi:error] [pid 10210:tid
>> 139829400610560] [client 127.0.0.1:45918] Timeout when reading response
>> headers from daemon process 'django_math':
>> /home/simamura/webapps/django_math/mathsite/mathsite/wsgi.py, referer:
>> https://www.google.com/
>> [Mon Oct 12 11:51:20.849576 2015] [wsgi:error] [pid 10210:tid
>> 139829283116800] [client
>>
>> ...
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To post to this group, send email to [email protected] <javascript:>
> .
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
