My main apache log level was set to warn, so I didn't see error messages which are having info level. After I changed it to debug, I found a lot of such lines in /var/log/apache2/error.log:
[Fri May 14 19:36:45.659418 2021] [wsgi:info] [pid 11806:tid 140271581906816] mod_wsgi (pid=11823): Process 'nm4-live' has died, d eregister and restart it. [Fri May 14 19:36:45.659990 2021] [wsgi:info] [pid 11806:tid 140271581906816] mod_wsgi (pid=11823): Process 'nm4-live' terminated by signal 9 [Fri May 14 19:36:45.660009 2021] [wsgi:info] [pid 11806:tid 140271581906816] mod_wsgi (pid=11823): Process 'nm4-live' has been de registered and will no longer be monitored. I found out that almost all memory of 32GB server RAM is resumed when apache processing requests. The old server had 256GB of RAM. New server has 32GB. I decreased the number of processes from 24 to 6 and the error disappeared. Thank you very much! I spent a week trying to find out the reason of the problem and lost any hope of solving it. Regards, Sergey пт, 14 мая 2021 г. в 03:55, Graham Dumpleton <[email protected]>: > Since you have LogLevel debug, mod_wsgi should be outputting messages > about the daemon process restarts and this can include from memory whether > was killed by certain signal. The information may be in the main Apache > error log rather than virtual host log. See if you can find them. > > Based on the message from uwsgi, I would speculate the reason is the same > for Apache. The message in uwsgi logs is: > > DAMN ! worker 17 (pid: 10800) died, killed by signal 9 :( trying respawn > ... > > As it is signal 9, this would tend to indicate that the Linux OOM reaper > felt the OS was running out of memory and decided for kill off the process > running your web application due to consuming a large amount of memory. > > Graham > > On 14 May 2021, at 5:40 am, Sergey <[email protected]> wrote: > > Hello, > > We need to move ubuntu 16.04/Apache + mod_wsgi + django 1.8/Nginx server > to new hardware. > I setup everything the same (apache + python + django app) as on old > ubuntu 16.04 server using docker container with ubuntu 16.04. nginx on host > is used as proxy. > New website is working, but some requests are failing with "Truncated or > oversized response headers received from daemon process" record in log. As > I understand wsgi processes are crashing and relaunched by apache (PIDs of > processes are changing after this error). If launch Django app in debug > mode without wsgi - everything works well. On old server everything is also > working well. WSGIApplicationGroup %{GLOBAL} is in config. Django app is > using modules which load C++ .so libraries. > > Server version: Apache/2.4.41 (Ubuntu) > Server built: 2020-04-13T17:19:17 > ---------- apache config for wsgi app: > LoadModule wsgi_module > /usr/local/lib/python2.7/dist-packages/mod_wsgi/server/mod_wsgi-py27.so > > Listen 127.0.0.1:8040 > > <VirtualHost 127.0.0.1:8040> > WSGIDaemonProcess nm4-live processes=24 threads=1 home=/app/app/ > python-path=/usr/local/lib/python2.7/dist-packages user=nm group=nm > display-name='wsgi:nm4-live' > WSGIProcessGroup nm4-live > WSGIApplicationGroup %{GLOBAL} > > WSGIScriptAlias / /app/app/nm4/wsgi_live.py > > LogLevel debug > CustomLog /app/log/apache/access.log "%h (%{X-Forwarded-For}i) %t > \"%r\" %>s (\"%{X-Original-Request}i\") %O %{sessionid}C" > ErrorLog /app/log/apache/error.log > > # There is streaming URL. Apache won't stream output if gzip is > enabled. > # Also not much sense to gzip traffic between apache and nginx. > # So just disable it for whole virtual host > SetEnv no-gzip > </VirtualHost> > > # Allow Apache to read wsgi.py files, otherwise it'll return "forbidden" > <Directory /app/app/nm4> > Order deny,allow > Require all granted > </Directory> > ---------------- > pip freeze: > cachetools==3.1.1 > certifi==2020.12.5 > cffi==1.14.5 > chardet==3.0.4 > click==7.1.2 > click-plugins==1.1.1 > cligj==0.7.1 > colour==0.1.5 > contextlib2==0.6.0.post1 > cryptography==3.3.2 > Django==1.8.19 > django-common-helpers==0.9.2 > django-cron==0.5.1 > django-jsonfield==0.9.15 > django-recaptcha==1.3.1 > djorm-pgarray==1.2 > docopt==0.4.0 > enum34==1.1.10 > Fiona==1.6.3 > google-api-python-client==1.7.3 > google-auth==1.30.0 > google-auth-httplib2==0.1.0 > httplib2==0.19.1 > idna==2.7 > ipaddress==1.0.23 > mailchimp==2.0.9 > mandrill==1.0.57 > markdown2==2.3.5 > mod-wsgi==4.7.1 > numexpr==2.7.3 > numpy==1.10.1 > oauth2client==4.1.2 > oauthlib==3.1.0 > pandas==0.17.0 > Pillow==5.2.0 > psycopg2-binary==2.8.6 > pyasn1==0.4.8 > pyasn1-modules==0.2.8 > pycparser==2.20 > PyJWT==1.7.1 > pymongo==2.8 > pyOpenSSL==20.0.1 > pyparsing==2.4.7 > Pyste==0.9.10 > python-dateutil==2.7.3 > python-memcached==1.59 > python-openid==2.2.5 > python-social-auth==0.2.21 > pytz==2018.5 > raven==6.9.0 > requests==2.19.1 > requests-oauthlib==1.3.0 > rsa==3.4.2 > Shapely==1.5.13 > six==1.16.0 > sorl-thumbnail==12.4.1 > stripe==1.35.0 > tables==3.4.4 > uritemplate==3.0.1 > urllib3==1.23 > uWSGI==2.0.19.1 > --- > python --version > Python 2.7.12 > --- > error in apache log: > > [Thu May 13 16:06:02.110200 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of Require all granted: granted, referer: ... > [Thu May 13 16:06:02.110238 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of <RequireAny>: granted, referer: ... > [Thu May 13 16:06:02.110298 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of Require all granted: granted, referer: ... > [Thu May 13 16:06:02.110307 2021] [authz_core:debug] [pid 1712:tid > 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: > authorization result of <RequireAny>: granted, referer: ... > [Thu May 13 16:06:36.186653 2021] [wsgi:error] [pid 1712:tid > 139677168236288] [client 127.0.0.1:38898] Truncated or oversized response > headers received from daemon process 'nm4-live': /app/app/nm4/wsgi_live.py, > referer: ... > -------- > As last resort I installed uwsgi for testing this situation and launched > it instead of apache: > --- > uwsgi --chdir=/app/app/ \ > --module=nm4.wsgi_live:application \ > --master --pidfile=/tmp/nm4-live-uwsgi-master.pid \ > --http-socket=127.0.0.1:8040 \ > --processes=25 \ > --uid=1000 --gid=1000 \ > --harakiri=20 \ > --max-requests=5000 \ > --vacuum \ > --stats :1717 \ > --daemonize=/app/log/uwsgi.log > --- > Now there is no such a mass of errors as with apache. > But if send a lot of requests uwsgi does not response for several of them > and this error is returned by nginx: 502 Bad Gateway. > In this case in uwsgi log I see this: > ---- > Thu May 13 19:08:01 2021 - HARAKIRI !!! end of worker 17 status !!! > DAMN ! worker 17 (pid: 10800) died, killed by signal 9 :( trying respawn > ... > Respawned uWSGI worker 17 (new pid: 10836) > ---- > New server has less CPU frequency. May be these harakiry is because django > processes request longer than 20 secs on high load (it makes > postgreSQL/postgis requests). > > Any help/hints will be appreciated. > > Regards, > Sergey. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/384288f2-e5a6-4490-9b19-38091687908fn%40googlegroups.com > <https://groups.google.com/d/msgid/modwsgi/384288f2-e5a6-4490-9b19-38091687908fn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "modwsgi" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/modwsgi/qOJF48JEuks/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/6BC57CE5-F43D-4208-BD60-2FF55319334E%40gmail.com > <https://groups.google.com/d/msgid/modwsgi/6BC57CE5-F43D-4208-BD60-2FF55319334E%40gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CALizj4b9iWM1pc6c4rF8qdFZw8qdXs6Wh8ZTptF-i9xGm5Fw0w%40mail.gmail.com.
