Everything was running along fine, then all of a sudden I received complaints that people were getting 403 forbidden from my server.
I looked in all the /var/log, server logs, etc., but didn't think to look at /tmp/mod_wsgi-localhost:8001:0/error_log until after restarting the service (I'm using systemd). I knew the 403 errors started at 8:21 Jan 21 by viewing the virtual host's request log, but after restarting the service found the following in the mod_wsgi log: [Mon Jan 20 21:53:13.249739 2020] [wsgi:error] [pid 20392:tid 140270704191232] [remote 198.199.82.162:35964] [2020-01-20 21:53:13,249] INFO in __init__: 66.249.79.119: GET http://routes.loutilities.com/robots.txt/routes 403 [Mon Jan 20 21:53:13.549214 2020] [wsgi:error] [pid 20392:tid 140270695798528] [remote 198.199.82.162:35968] [2020-01-20 21:53:13,549] INFO in __init__: 66.249.79.121: GET http://routes.loutilities.com/fsrc/routes 200 [Tue Jan 21 04:47:53.399163 2020] [wsgi:error] [pid 20392:tid 140270704191232] [remote 198.199.82.162:42670] [2020-01-21 04:47:53,398] INFO in __init__: 173.252.87.47: GET http://routes.loutilities.com/ 302 [Tue Jan 21 04:47:55.573795 2020] [wsgi:error] [pid 20392:tid 140270695798528] [remote 198.199.82.162:42674] [2020-01-21 04:47:55,573] INFO in __init__: 173.252.95.26: GET http://routes.loutilities.com/ 302 [Tue Jan 21 08:21:05.017486 2020] [authz_core:error] [pid 20393:tid 140271178082048] [client 198.199.82.162:49262] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //steeplechasers.org/training/running-routes/ [Tue Jan 21 08:21:09.020530 2020] [authz_core:error] [pid 20420:tid 140271178348288] [client 198.199.82.162:49264] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //steeplechasers.org/training/running-routes/ [Tue Jan 21 08:21:09.952862 2020] [authz_core:error] [pid 20393:tid 140271099369216] [client 198.199.82.162:49266] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //steeplechasers.org/training/running-routes/ [Tue Jan 21 08:21:10.811980 2020] [authz_core:error] [pid 20420:tid 140271178082048] [client 198.199.82.162:49268] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //steeplechasers.org/training/running-routes/ [Tue Jan 21 08:21:19.475125 2020] [authz_core:error] [pid 20393:tid 140271177549568] [client 198.199.82.162:49270] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //steeplechasers.org/training/running-routes/ [Tue Jan 21 08:21:23.131160 2020] [authz_core:error] [pid 20420:tid 140271178614528] [client 198.199.82.162:49272] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //steeplechasers.org/training/running-routes/ [Tue Jan 21 08:21:24.440572 2020] [authz_core:error] [pid 20420:tid 140271177549568] [client 198.199.82.162:49276] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //steeplechasers.org/training/running-routes/ [Tue Jan 21 08:23:38.045572 2020] [authz_core:error] [pid 20420:tid 140271099369216] [client 198.199.82.162:49322] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs [Tue Jan 21 08:36:40.705930 2020] [authz_core:error] [pid 20393:tid 140271098836736] [client 198.199.82.162:49516] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //m.facebook.com/ [Tue Jan 21 08:36:41.109523 2020] [authz_core:error] [pid 20393:tid 140271177283328] [client 198.199.82.162:49518] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: https: //routes.loutilities.com/fsrc/routes [Tue Jan 21 08:47:53.231357 2020] [authz_core:error] [pid 20420:tid 140271098836736] [client 198.199.82.162:49730] AH01630: client denied by server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs i.e., everything running along fine until the request 8:21. As mentioned I didn't look into this log until after restarting the service, so I didn't get a chance to inspect the /tmp/mod_wsgi-localhost: 8001:0/htdocs directory to see what the issue was. I did systemctl status on the service before the restart and all looked well. [lking@loutility-server-digitalocean ~]$ sudo systemctl status vhost- routetility-sandbox.service ● vhost-routetility-sandbox.service - start routetility www proxy server Loaded: loaded (/etc/systemd/system/vhost-routetility-sandbox.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2020-01-20 11:53:57 EST; 23h ago Main PID: 19331 (start-wsgi-serv) CGroup: /system.slice/vhost-routetility-sandbox.service ├─19331 /bin/bash /root/bin/start-wsgi-server runningroutes sandbox.routes.loutilities.com routesmgr routesmgr 8002 ├─19332 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8002: 0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... ├─19335 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8002: 0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... ├─19978 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost:8002: 0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... └─19979 (wsgi:localhost:8002:0) -f /tmp/mod_wsgi-localhost:8002: 0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Server Conf : /tmp/mod_wsgi-localhost:8002:0/ht...onf Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Error Log File : /tmp/mod_wsgi-localhost:8002:0/er...rn) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Request Capacity : 5 (1 process * 5 threads) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Request Timeout : 60 (seconds) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Startup Timeout : 15 (seconds) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Queue Backlog : 100 (connections) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Queue Timeout : 45 (seconds) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Server Capacity : 20 (event/worker), 20 (prefork) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Server Backlog : 500 (connections) Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: Locale Setting : en_US.UTF-8 Hint: Some lines were ellipsized, use -l to show in full. Have you heard of this before? I will definitely check /tmp/mod_wsgi- localhost:8001:0/htdocs directory if this happens again. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/aaa686b3-265c-44ad-8604-95551cc35d31%40googlegroups.com.
