and for completeness [lking@loutility-server-digitalocean applogs]$ sudo cat /root/bin/vhost- routetility-www #!/bin/bash source /var/www/www.routes.loutilities.com/venv/bin/activate mod_wsgi-express start-server --server-name proxysvr.loutilities.com --port 8001 --user routesmgr --group routesmgr /var/www/www.routes.loutilities.com/ runningroutes/runningroutes/runningroutes.wsgi --working-directory /var/www/ www.routes.loutilities.com/runningroutes/runningroutes/ deactivate
On Tuesday, January 21, 2020 at 12:54:43 PM UTC-5, Lou King wrote: > > Oops, posted the wrong servics log, sorry it should have been > > [lking@loutility-server-digitalocean ~]$ sudo systemctl status vhost- > routetility-www.service > ● vhost-routetility-www.service - start routetility www proxy server > Loaded: loaded (/etc/systemd/system/vhost-routetility-www.service; > enabled; vendor preset: disabled) > Active: active (running) since Mon 2020-01-20 12:11:29 EST; 23h ago > Main PID: 20389 (vhost-routetili) > CGroup: /system.slice/vhost-routetility-www.service > ├─20389 /bin/bash /root/bin/vhost-routetility-www > ├─20390 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost: > 8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... > ├─20392 (wsgi:localhost:8001:0) -f /tmp/mod_wsgi-localhost: > 8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... > ├─20393 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost: > 8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... > └─20420 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost: > 8001:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... > > > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Server Conf : /tmp/mod_wsgi-localhost:8001:...onf > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Error Log File : /tmp/mod_wsgi-localhost:8001:...rn) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Request Capacity : 5 (1 process * 5 threads) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Request Timeout : 60 (seconds) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Startup Timeout : 15 (seconds) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Queue Backlog : 100 (connections) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Queue Timeout : 45 (seconds) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Server Capacity : 20 (event/worker), 20 (prefork) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Server Backlog : 500 (connections) > Jan 20 12:11:29 loutility-server-digitalocean vhost-routetility-www[20389 > ]: Locale Setting : en_US.UTF-8 > Hint: Some lines were ellipsized, use -l to show in full. > > > > > On Tuesday, January 21, 2020 at 12:52:48 PM UTC-5, Lou King wrote: >> >> Everything was running along fine, then all of a sudden I received >> complaints that people were getting 403 forbidden from my server. >> >> I looked in all the /var/log, server logs, etc., but didn't think to look >> at /tmp/mod_wsgi-localhost:8001:0/error_log until after restarting the >> service (I'm using systemd). I knew the 403 errors started at 8:21 Jan 21 >> by viewing the virtual host's request log, but after restarting the service >> found the following in the mod_wsgi log: >> >> [Mon Jan 20 21:53:13.249739 2020] [wsgi:error] [pid 20392:tid >> 140270704191232] [remote 198.199.82.162:35964] [2020-01-20 21:53:13,249] >> INFO in __init__: 66.249.79.119: GET http:// >> routes.loutilities.com/robots.txt/routes 403 >> [Mon Jan 20 21:53:13.549214 2020] [wsgi:error] [pid 20392:tid >> 140270695798528] [remote 198.199.82.162:35968] [2020-01-20 21:53:13,549] >> INFO in __init__: 66.249.79.121: GET http:// >> routes.loutilities.com/fsrc/routes 200 >> [Tue Jan 21 04:47:53.399163 2020] [wsgi:error] [pid 20392:tid >> 140270704191232] [remote 198.199.82.162:42670] [2020-01-21 04:47:53,398] >> INFO in __init__: 173.252.87.47: GET http://routes.loutilities.com/ 302 >> [Tue Jan 21 04:47:55.573795 2020] [wsgi:error] [pid 20392:tid >> 140270695798528] [remote 198.199.82.162:42674] [2020-01-21 04:47:55,573] >> INFO in __init__: 173.252.95.26: GET http://routes.loutilities.com/ 302 >> [Tue Jan 21 08:21:05.017486 2020] [authz_core:error] [pid 20393:tid >> 140271178082048] [client 198.199.82.162:49262] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://steeplechasers.org/training/running-routes/ >> [Tue Jan 21 08:21:09.020530 2020] [authz_core:error] [pid 20420:tid >> 140271178348288] [client 198.199.82.162:49264] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://steeplechasers.org/training/running-routes/ >> [Tue Jan 21 08:21:09.952862 2020] [authz_core:error] [pid 20393:tid >> 140271099369216] [client 198.199.82.162:49266] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://steeplechasers.org/training/running-routes/ >> [Tue Jan 21 08:21:10.811980 2020] [authz_core:error] [pid 20420:tid >> 140271178082048] [client 198.199.82.162:49268] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://steeplechasers.org/training/running-routes/ >> [Tue Jan 21 08:21:19.475125 2020] [authz_core:error] [pid 20393:tid >> 140271177549568] [client 198.199.82.162:49270] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://steeplechasers.org/training/running-routes/ >> [Tue Jan 21 08:21:23.131160 2020] [authz_core:error] [pid 20420:tid >> 140271178614528] [client 198.199.82.162:49272] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://steeplechasers.org/training/running-routes/ >> [Tue Jan 21 08:21:24.440572 2020] [authz_core:error] [pid 20420:tid >> 140271177549568] [client 198.199.82.162:49276] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://steeplechasers.org/training/running-routes/ >> [Tue Jan 21 08:23:38.045572 2020] [authz_core:error] [pid 20420:tid >> 140271099369216] [client 198.199.82.162:49322] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs >> [Tue Jan 21 08:36:40.705930 2020] [authz_core:error] [pid 20393:tid >> 140271098836736] [client 198.199.82.162:49516] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://m.facebook.com/ >> [Tue Jan 21 08:36:41.109523 2020] [authz_core:error] [pid 20393:tid >> 140271177283328] [client 198.199.82.162:49518] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs, referer: >> https://routes.loutilities.com/fsrc/routes >> [Tue Jan 21 08:47:53.231357 2020] [authz_core:error] [pid 20420:tid >> 140271098836736] [client 198.199.82.162:49730] AH01630: client denied by >> server configuration: /tmp/mod_wsgi-localhost:8001:0/htdocs >> >> >> i.e., everything running along fine until the request 8:21. >> >> As mentioned I didn't look into this log until after restarting the >> service, so I didn't get a chance to inspect the /tmp/mod_wsgi-localhost: >> 8001:0/htdocs directory to see what the issue was. >> >> I did systemctl status on the service before the restart and all looked >> well. >> >> [lking@loutility-server-digitalocean ~]$ sudo systemctl status vhost- >> routetility-sandbox.service >> ● vhost-routetility-sandbox.service - start routetility www proxy server >> Loaded: loaded (/etc/systemd/system/vhost-routetility-sandbox.service; >> enabled; vendor preset: disabled) >> Active: active (running) since Mon 2020-01-20 11:53:57 EST; 23h ago >> Main PID: 19331 (start-wsgi-serv) >> CGroup: /system.slice/vhost-routetility-sandbox.service >> ├─19331 /bin/bash /root/bin/start-wsgi-server runningroutes >> sandbox.routes.loutilities.com routesmgr routesmgr 8002 >> ├─19332 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost: >> 8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... >> ├─19335 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost: >> 8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... >> ├─19978 httpd (mod_wsgi-express) -f /tmp/mod_wsgi-localhost: >> 8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... >> └─19979 (wsgi:localhost:8002:0) -f /tmp/mod_wsgi-localhost: >> 8002:0/httpd.conf -DMOD_WSGI_VIRTUAL_HOST -DMOD_WSGI_MPM_E... >> >> >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Server Conf : /tmp/mod_wsgi-localhost:8002:0/ht...onf >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Error Log File : /tmp/mod_wsgi-localhost:8002:0/er...rn) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Request Capacity : 5 (1 process * 5 threads) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Request Timeout : 60 (seconds) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Startup Timeout : 15 (seconds) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Queue Backlog : 100 (connections) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Queue Timeout : 45 (seconds) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Server Capacity : 20 (event/worker), 20 (prefork) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Server Backlog : 500 (connections) >> Jan 20 11:53:57 loutility-server-digitalocean start-wsgi-server[19331]: >> Locale Setting : en_US.UTF-8 >> Hint: Some lines were ellipsized, use -l to show in full. >> >> >> Have you heard of this before? I will definitely check /tmp/mod_wsgi- >> localhost:8001:0/htdocs directory if this happens again. >> >> -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/5a656c80-c2be-47aa-b893-0a4b61de5f0f%40googlegroups.com.
