Hello,
We need to move ubuntu 16.04/Apache + mod_wsgi + django 1.8/Nginx server to
new hardware.
I setup everything the same (apache + python + django app) as on old ubuntu
16.04 server using docker container with ubuntu 16.04. nginx on host is
used as proxy.
New website is working, but some requests are failing with "Truncated or
oversized response headers received from daemon process" record in log. As
I understand wsgi processes are crashing and relaunched by apache (PIDs of
processes are changing after this error). If launch Django app in debug
mode without wsgi - everything works well. On old server everything is also
working well. WSGIApplicationGroup %{GLOBAL} is in config. Django app is
using modules which load C++ .so libraries.
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2020-04-13T17:19:17
---------- apache config for wsgi app:
LoadModule wsgi_module
/usr/local/lib/python2.7/dist-packages/mod_wsgi/server/mod_wsgi-py27.so
Listen 127.0.0.1:8040
<VirtualHost 127.0.0.1:8040>
WSGIDaemonProcess nm4-live processes=24 threads=1 home=/app/app/
python-path=/usr/local/lib/python2.7/dist-packages user=nm group=nm
display-name='wsgi:nm4-live'
WSGIProcessGroup nm4-live
WSGIApplicationGroup %{GLOBAL}
WSGIScriptAlias / /app/app/nm4/wsgi_live.py
LogLevel debug
CustomLog /app/log/apache/access.log "%h (%{X-Forwarded-For}i) %t
\"%r\" %>s (\"%{X-Original-Request}i\") %O %{sessionid}C"
ErrorLog /app/log/apache/error.log
# There is streaming URL. Apache won't stream output if gzip is enabled.
# Also not much sense to gzip traffic between apache and nginx.
# So just disable it for whole virtual host
SetEnv no-gzip
</VirtualHost>
# Allow Apache to read wsgi.py files, otherwise it'll return "forbidden"
<Directory /app/app/nm4>
Order deny,allow
Require all granted
</Directory>
----------------
pip freeze:
cachetools==3.1.1
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==7.1.2
click-plugins==1.1.1
cligj==0.7.1
colour==0.1.5
contextlib2==0.6.0.post1
cryptography==3.3.2
Django==1.8.19
django-common-helpers==0.9.2
django-cron==0.5.1
django-jsonfield==0.9.15
django-recaptcha==1.3.1
djorm-pgarray==1.2
docopt==0.4.0
enum34==1.1.10
Fiona==1.6.3
google-api-python-client==1.7.3
google-auth==1.30.0
google-auth-httplib2==0.1.0
httplib2==0.19.1
idna==2.7
ipaddress==1.0.23
mailchimp==2.0.9
mandrill==1.0.57
markdown2==2.3.5
mod-wsgi==4.7.1
numexpr==2.7.3
numpy==1.10.1
oauth2client==4.1.2
oauthlib==3.1.0
pandas==0.17.0
Pillow==5.2.0
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
PyJWT==1.7.1
pymongo==2.8
pyOpenSSL==20.0.1
pyparsing==2.4.7
Pyste==0.9.10
python-dateutil==2.7.3
python-memcached==1.59
python-openid==2.2.5
python-social-auth==0.2.21
pytz==2018.5
raven==6.9.0
requests==2.19.1
requests-oauthlib==1.3.0
rsa==3.4.2
Shapely==1.5.13
six==1.16.0
sorl-thumbnail==12.4.1
stripe==1.35.0
tables==3.4.4
uritemplate==3.0.1
urllib3==1.23
uWSGI==2.0.19.1
---
python --version
Python 2.7.12
---
error in apache log:
[Thu May 13 16:06:02.110200 2021] [authz_core:debug] [pid 1712:tid
139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626:
authorization result of Require all granted: granted, referer: ...
[Thu May 13 16:06:02.110238 2021] [authz_core:debug] [pid 1712:tid
139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626:
authorization result of <RequireAny>: granted, referer: ...
[Thu May 13 16:06:02.110298 2021] [authz_core:debug] [pid 1712:tid
139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626:
authorization result of Require all granted: granted, referer: ...
[Thu May 13 16:06:02.110307 2021] [authz_core:debug] [pid 1712:tid
139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626:
authorization result of <RequireAny>: granted, referer: ...
[Thu May 13 16:06:36.186653 2021] [wsgi:error] [pid 1712:tid
139677168236288] [client 127.0.0.1:38898] Truncated or oversized response
headers received from daemon process 'nm4-live': /app/app/nm4/wsgi_live.py,
referer: ...
--------
As last resort I installed uwsgi for testing this situation and launched it
instead of apache:
---
uwsgi --chdir=/app/app/ \
--module=nm4.wsgi_live:application \
--master --pidfile=/tmp/nm4-live-uwsgi-master.pid \
--http-socket=127.0.0.1:8040 \
--processes=25 \
--uid=1000 --gid=1000 \
--harakiri=20 \
--max-requests=5000 \
--vacuum \
--stats :1717 \
--daemonize=/app/log/uwsgi.log
---
Now there is no such a mass of errors as with apache.
But if send a lot of requests uwsgi does not response for several of them
and this error is returned by nginx: 502 Bad Gateway.
In this case in uwsgi log I see this:
----
Thu May 13 19:08:01 2021 - HARAKIRI !!! end of worker 17 status !!!
DAMN ! worker 17 (pid: 10800) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 17 (new pid: 10836)
----
New server has less CPU frequency. May be these harakiry is because django
processes request longer than 20 secs on high load (it makes
postgreSQL/postgis requests).
Any help/hints will be appreciated.
Regards,
Sergey.
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/384288f2-e5a6-4490-9b19-38091687908fn%40googlegroups.com.
