Since you have LogLevel debug, mod_wsgi should be outputting messages about the daemon process restarts and this can include from memory whether was killed by certain signal. The information may be in the main Apache error log rather than virtual host log. See if you can find them.
Based on the message from uwsgi, I would speculate the reason is the same for Apache. The message in uwsgi logs is: DAMN ! worker 17 (pid: 10800) died, killed by signal 9 :( trying respawn ... As it is signal 9, this would tend to indicate that the Linux OOM reaper felt the OS was running out of memory and decided for kill off the process running your web application due to consuming a large amount of memory. Graham > On 14 May 2021, at 5:40 am, Sergey <[email protected]> wrote: > > Hello, > > We need to move ubuntu 16.04/Apache + mod_wsgi + django 1.8/Nginx server to > new hardware. > I setup everything the same (apache + python + django app) as on old ubuntu > 16.04 server using docker container with ubuntu 16.04. nginx on host is used > as proxy. > New website is working, but some requests are failing with "Truncated or > oversized response headers received from daemon process" record in log. As I > understand wsgi processes are crashing and relaunched by apache (PIDs of > processes are changing after this error). If launch Django app in debug mode > without wsgi - everything works well. On old server everything is also > working well. WSGIApplicationGroup %{GLOBAL} is in config. Django app is > using modules which load C++ .so libraries. > > Server version: Apache/2.4.41 (Ubuntu) > Server built: 2020-04-13T17:19:17 > ---------- apache config for wsgi app: > LoadModule wsgi_module > /usr/local/lib/python2.7/dist-packages/mod_wsgi/server/mod_wsgi-py27.so > > Listen 127.0.0.1:8040 > > <VirtualHost 127.0.0.1:8040> > WSGIDaemonProcess nm4-live processes=24 threads=1 home=/app/app/ > python-path=/usr/local/lib/python2.7/dist-packages user=nm group=nm > display-name='wsgi:nm4-live' > WSGIProcessGroup nm4-live > WSGIApplicationGroup %{GLOBAL} > > WSGIScriptAlias / /app/app/nm4/wsgi_live.py > > LogLevel debug > CustomLog /app/log/apache/access.log "%h (%{X-Forwarded-For}i) %t \"%r\" > %>s (\"%{X-Original-Request}i\") %O %{sessionid}C" > ErrorLog /app/log/apache/error.log > > # There is streaming URL. Apache won't stream output if gzip is enabled. > # Also not much sense to gzip traffic between apache and nginx. > # So just disable it for whole virtual host > SetEnv no-gzip > </VirtualHost> > > # Allow Apache to read wsgi.py files, otherwise it'll return "forbidden" > <Directory /app/app/nm4> > Order deny,allow > Require all granted > </Directory> > ---------------- > pip freeze: > cachetools==3.1.1 > certifi==2020.12.5 > cffi==1.14.5 > chardet==3.0.4 > click==7.1.2 > click-plugins==1.1.1 > cligj==0.7.1 > colour==0.1.5 > contextlib2==0.6.0.post1 > cryptography==3.3.2 > Django==1.8.19 > django-common-helpers==0.9.2 > django-cron==0.5.1 > django-jsonfield==0.9.15 > django-recaptcha==1.3.1 > djorm-pgarray==1.2 > docopt==0.4.0 > enum34==1.1.10 > Fiona==1.6.3 > google-api-python-client==1.7.3 > google-auth==1.30.0 > google-auth-httplib2==0.1.0 > httplib2==0.19.1 > idna==2.7 > ipaddress==1.0.23 > mailchimp==2.0.9 > mandrill==1.0.57 > markdown2==2.3.5 > mod-wsgi==4.7.1 > numexpr==2.7.3 > numpy==1.10.1 > oauth2client==4.1.2 > oauthlib==3.1.0 > pandas==0.17.0 > Pillow==5.2.0 > psycopg2-binary==2.8.6 > pyasn1==0.4.8 > pyasn1-modules==0.2.8 > pycparser==2.20 > PyJWT==1.7.1 > pymongo==2.8 > pyOpenSSL==20.0.1 > pyparsing==2.4.7 > Pyste==0.9.10 > python-dateutil==2.7.3 > python-memcached==1.59 > python-openid==2.2.5 > python-social-auth==0.2.21 > pytz==2018.5 > raven==6.9.0 > requests==2.19.1 > requests-oauthlib==1.3.0 > rsa==3.4.2 > Shapely==1.5.13 > six==1.16.0 > sorl-thumbnail==12.4.1 > stripe==1.35.0 > tables==3.4.4 > uritemplate==3.0.1 > urllib3==1.23 > uWSGI==2.0.19.1 > --- > python --version > Python 2.7.12 > --- > error in apache log: > > [Thu May 13 16:06:02.110200 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of Require all granted: granted, referer: ... > [Thu May 13 16:06:02.110238 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of <RequireAny>: granted, referer: ... > [Thu May 13 16:06:02.110298 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of Require all granted: granted, referer: ... > [Thu May 13 16:06:02.110307 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of <RequireAny>: granted, referer: ... > [Thu May 13 16:06:36.186653 2021] [wsgi:error] [pid 1712:tid 139677168236288] > [client 127.0.0.1:38898] Truncated or oversized response headers received > from daemon process 'nm4-live': /app/app/nm4/wsgi_live.py, referer: ... > -------- > As last resort I installed uwsgi for testing this situation and launched it > instead of apache: > --- > uwsgi --chdir=/app/app/ \ > --module=nm4.wsgi_live:application \ > --master --pidfile=/tmp/nm4-live-uwsgi-master.pid \ > --http-socket=127.0.0.1:8040 \ > --processes=25 \ > --uid=1000 --gid=1000 \ > --harakiri=20 \ > --max-requests=5000 \ > --vacuum \ > --stats :1717 \ > --daemonize=/app/log/uwsgi.log > --- > Now there is no such a mass of errors as with apache. > But if send a lot of requests uwsgi does not response for several of them and > this error is returned by nginx: 502 Bad Gateway. > In this case in uwsgi log I see this: > ---- > Thu May 13 19:08:01 2021 - HARAKIRI !!! end of worker 17 status !!! > DAMN ! worker 17 (pid: 10800) died, killed by signal 9 :( trying respawn ... > Respawned uWSGI worker 17 (new pid: 10836) > ---- > New server has less CPU frequency. May be these harakiry is because django > processes request longer than 20 secs on high load (it makes > postgreSQL/postgis requests). > > Any help/hints will be appreciated. > > Regards, > Sergey. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/384288f2-e5a6-4490-9b19-38091687908fn%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/384288f2-e5a6-4490-9b19-38091687908fn%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/6BC57CE5-F43D-4208-BD60-2FF55319334E%40gmail.com.
