Graham, It does click now. Couple of months ago, I had to modify the original wsgi.py file to include Environmental variables read from my Virtual Host.
__________________________________________________________________________ *I changed from: * *On wsgi.py* import os from django.core.wsgi import get_wsgi_application os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings') application = get_wsgi_application() *to* *On Virtual Host* SetEnv VAR1 xxxx SetEnv VAR2 yyyy *On wsgi.py* import os from django.core.wsgi import get_wsgi_application os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings') def application(environ, start_response): os.environ['VAR1'] = environ.get('VAR1', '') os.environ['VAR2'] = environ.get('VAR2', '') _application = get_wsgi_application() return _application(environ, start_response) __________________________________________________________________________ I think I found this solution on a forum and it worked and never expected that it would yield on such consequences. Do you have any suggestion on the right way to do this ? I remember testing multiple options and this was the only one that worked. Thanks, Juan Khawly On Friday, September 9, 2022 at 4:46:42 PM UTC-4 Graham Dumpleton wrote: > As logs show, you have a problem with thread locking related to logging > subsystem of Python. > > What do you have in your wsgi.py file? > > The messages suggest you are calling Django's get_wsgi_application() on > every request, which is a bad idea. It should only be called once at top > level scope in wsgi.py, not in a request handler function. > > Graham > > On 10 Sep 2022, at 2:41 am, Juan Khawly <juan...@gmail.com> wrote: > > Graham, > > After adding the timeout, as you said, the server auto recovers from the > problem. > > After mod_wsgi is logging info level. My error log now gives traces of > where the problem is. I'm attaching my error.log from today (sep 09), any > ideas? > > Thanks for the support. > Juan Khawly > > On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote: > >> Makes total sense. >> >> Just added the option to the DaemonProcess and LogLevel info to the >> virtual host. I will be monitoring the the logs and report back in a couple >> days for reference. >> >> Appreciate your help. >> Juan Khawly >> >> On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton >> wrote: >> >>> >>> On 7 Sep 2022, at 11:43 pm, Juan Khawly <juan...@gmail.com> wrote: >>> >>> Graham, >>> >>> Going to make that change, monitor and keep this chat updated with the >>> result. >>> >>> 2 Questions: >>> >>> 1) The option request-timeout = 60 is included inside the virtual host >>> along with the rest of the Daemon code right ? >>> >>> >>> It is an option to be added to the existing WSGIDaemonProcess directive. >>> >>> 2) *Under no traffic*, do you have any idea of why this problem could >>> happen? As I explained, it is usually, but not always, preceded by couple >>> of GET Request from a random IP (bot requests) to random urls. My >>> assumption was Slow DDOS and this is why I enabled modreqtimeout, mod >>> security and mod qos. But at this point I'm clueless of how to diagnose. >>> >>> >>> No idea. If it was truly a slow DDOS attack the request wouldn't >>> actually show in the access logs because Apache only logs requests on >>> completion. So am not sure one could say is related to those other >>> requests. I would say it is more likely that over time a trickle of >>> requests come in to your application as normal which block and slowly use >>> up capacity. Hopefully the stack trace created when get a forced restart >>> due to request timeout will show where. Just keep in mind that since the >>> request timeout will cause auto recovery you may not notice it occurred, so >>> you will need to periodically check Apache error logs yourself. Make sure >>> that have info LogLevel for the virtual host so get more useful information >>> out of mod_wsgi. >>> >>> Thanks >>> Juan Khawly >>> >>> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote: >>> >>>> Sorry, seems I didn't see your update. >>>> >>>> Add an option: >>>> >>>> request-timeout=60 >>>> >>>> to the WSGIDaemonProcess. >>>> >>>> Set the value (in seconds) greater than you would expect your HTTP >>>> requests to normally run. >>>> >>>> What will happen is that when the average running time for all possible >>>> concurrent requests exceeds that timeout value, the daemon process will be >>>> forcibly restarted. This will have the effect of unblocking the process >>>> and >>>> a new one will be started in its place. So acts as a fail safe to ensure >>>> your application keeps running. >>>> >>>> What this will also do is attempt to dump out Python stack traces for >>>> what all the request handler threads were doing when the process is >>>> restarted. This will hopefully allow you to work out why your request >>>> handlers are getting stuck, be it they are getting stuck on a lock, or >>>> waiting on a backend service. >>>> >>>> In short, your request handlers are getting stuck and not completing. >>>> Over time these are building up and the thread pool for handling requests >>>> is exhausted and so the process stops handling requests. >>>> >>>> Graham >>>> >>>> On 7 Sep 2022, at 5:44 am, Juan Khawly <juan...@gmail.com> wrote: >>>> >>>> Any ideas? >>>> >>>> Thanks >>>> >>>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote: >>>> >>>>> Hello Graham, >>>>> >>>>> I'm going to try to address your questions: >>>>> >>>>> *Inside my Virtual Host* >>>>> >>>>> Alias /static /data/home/user/project/frontend/build/static >>>>> <Directory /data/home/user/project/frontend/build/static> >>>>> Require all granted >>>>> </Directory> >>>>> >>>>> <Directory /data/home/user/project/my_project> >>>>> <Files wsgi.py> >>>>> Require all granted >>>>> </Files> >>>>> </Directory> >>>>> >>>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py >>>>> WSGIDaemonProcess my_project >>>>> python-path=/data/home/user/project >>>>> python-home=/data/home/user/environment/venv >>>>> WSGIProcessGroup my_project >>>>> >>>>> >>>>> *Inside apache2.conf* >>>>> >>>>> WSGIApplicationGroup %{GLOBAL} >>>>> >>>>> *On the apache/error.log* >>>>> When I get the 503 on the access.log, these are the types of errors >>>>> seen on the error.log >>>>> >>>>> *One type of error* >>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading >>>>> response headers from daemon process 'my_project': >>>>> /data/home/project/my_project/my_project/wsgi.py >>>>> >>>>> *Another type of error* >>>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid >>>>> 140518595991296] (11)Resource temporarily unavailable: [client >>>>> 172.31.17.102:31880] mod_wsgi (pid=3267): Unable to connect to WSGI >>>>> daemon process ' my_project ' on '/var/run/apache2/wsgi.2385.1.1.sock' >>>>> after multiple attempts as listener backlog limit was exceeded or the >>>>> socket does not exist. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> Juan Khawly >>>>> >>>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton >>>>> wrote: >>>>> >>>>>> Would need to see the mod_wsgi configuration you are using to >>>>>> configure the WSGI application, including how WSGIDaemonProcess is >>>>>> configured and whether you are using WSGIApplicationGroup. Also, what >>>>>> errors are in the Apache error log when the 503 errors occur. >>>>>> >>>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@gmail.com> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I've been running into this problem for a while. >>>>>> >>>>>> *CONTEXT * >>>>>> >>>>>> I have an application developed in python (3.10), django 4.0.3, using >>>>>> mod_wsgi and apache. The application is in a DEV environment and hosted >>>>>> in >>>>>> AWS EC2. Currently, it does not receive traffic at all. >>>>>> >>>>>> *Installation of Mod WSGI* >>>>>> apt-get install -y apache2-dev >>>>>> >>>>>> *Setup out of the VENV* >>>>>> mod_wsgi-express install-module >>>>>> >>>>>> editing: /etc/apache2/mods-available/wsgi.load >>>>>> >>>>>> LoadModule wsgi_module "/usr/lib/apache2/modules/ >>>>>> mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so" >>>>>> WSGIPythonHome "/data/home/user/environment/venv" >>>>>> >>>>>> *Module Enabled* >>>>>> a2enmod wsgi >>>>>> >>>>>> *PROBLEM* >>>>>> >>>>>> The application works perfect most of the time. Couple of times a >>>>>> week, without traffic the apache server goes down into 503. Usually it >>>>>> is >>>>>> preceded by a random request but it does not always happen that way. I >>>>>> am >>>>>> assuming that is Slow DDOS but I want to make sure it is not miss >>>>>> configuration of the WSGI. >>>>>> >>>>>> access.log example >>>>>> >>>>>> <access.PNG> >>>>>> >>>>>> error.log example >>>>>> I masked the internal routes >>>>>> >>>>>> *This is one of the errors:* >>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading >>>>>> response headers from daemon process 'XXXXX': >>>>>> /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py >>>>>> >>>>>> *Another type of error:* >>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid >>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading >>>>>> response headers from daemon process 'XXXXXXX': >>>>>> /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py >>>>>> >>>>>> *SOLUTION* >>>>>> >>>>>> If I restart the server, all works again until next failure. >>>>>> >>>>>> I've enabled the following modules, in case it is SlowDDOS >>>>>> modreqtimeout >>>>>> libapache2-mod-qos >>>>>> libapache2-mod-security2. >>>>>> >>>>>> Any recommendation? >>>>>> >>>>>> Thanks, >>>>>> Juan Khawly >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "modwsgi" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to modwsgi+u...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> <access.PNG> >>>>>> >>>>>> >>>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to modwsgi+u...@googlegroups.com. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to modwsgi+u...@googlegroups.com. >>> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to modwsgi+u...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > <error.log> > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com.