Graham,
If I take the the get_wsgi_application() out of the function, my site
throws error an internal error. Looking inside the logs, it all comes down
that my app is not receiving the value of VAR1 or VAR2 that are further
used in my settings.py to setup the environment. VAR1 and VAR2 are
environment variables that help me recognize if I am in LOCAL, DEV or PROD.
Also, I honestly didn't understand this comment if you can help me see more
clearly. *"Be aware that what you are doing is not technically safe because
environment variables are global and so if different URLs ended up in
different values for the variables, then a different request could change
it before you got to use it." *
My goal is having environmental variables inside my virtual host and pass
them to the Django app. I tried this using the environment variables files
that Linux uses but I was never successful to pass them to my Django app.
Thanks,
Juan Khawly
On Friday, September 9, 2022 at 9:57:06 PM UTC-4 Graham Dumpleton wrote:
> Be aware that what you are doing is not technically safe because
> environment variables are global and so if different URLs ended up in
> different values for the variables, then a different request could change
> it before you got to use it.
>
> That said, use:
>
> import os
> from django.core.wsgi import get_wsgi_application
> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
>
> _application = get_wsgi_application()
>
> def application(environ, start_response):
> os.environ['VAR1'] = environ.get('VAR1', '')
> os.environ['VAR2'] = environ.get('VAR2', '')
> return _application(environ, start_response)
>
> In other words, just call get_wsgi_application() once outside of the
> function.
>
> Graham
>
> On 10 Sep 2022, at 11:36 am, Juan Khawly <juan...@gmail.com> wrote:
>
> Graham,
>
> It does click now. Couple of months ago, I had to modify the original
> wsgi.py file to include Environmental variables read from my Virtual Host.
>
> __________________________________________________________________________
> *I changed from: *
>
> *On wsgi.py*
>
> import os
> from django.core.wsgi import get_wsgi_application
> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
> application = get_wsgi_application()
>
> *to*
>
> *On Virtual Host*
>
> SetEnv VAR1 xxxx
> SetEnv VAR2 yyyy
>
> *On wsgi.py*
>
> import os
> from django.core.wsgi import get_wsgi_application
> os.environ.setdefault('DJANGO_SETTINGS_MODULE', ' project .settings')
>
> def application(environ, start_response):
> os.environ['VAR1'] = environ.get('VAR1', '')
> os.environ['VAR2'] = environ.get('VAR2', '')
> _application = get_wsgi_application()
> return _application(environ, start_response)
> __________________________________________________________________________
>
> I think I found this solution on a forum and it worked and never expected
> that it would yield on such consequences.
>
> Do you have any suggestion on the right way to do this ? I remember
> testing multiple options and this was the only one that worked.
>
> Thanks,
> Juan Khawly
>
>
>
>
>
>
>
>
>
> On Friday, September 9, 2022 at 4:46:42 PM UTC-4 Graham Dumpleton wrote:
>
>> As logs show, you have a problem with thread locking related to logging
>> subsystem of Python.
>>
>> What do you have in your wsgi.py file?
>>
>> The messages suggest you are calling Django's get_wsgi_application() on
>> every request, which is a bad idea. It should only be called once at top
>> level scope in wsgi.py, not in a request handler function.
>>
>> Graham
>>
>> On 10 Sep 2022, at 2:41 am, Juan Khawly <juan...@gmail.com> wrote:
>>
>> Graham,
>>
>> After adding the timeout, as you said, the server auto recovers from the
>> problem.
>>
>> After mod_wsgi is logging info level. My error log now gives traces of
>> where the problem is. I'm attaching my error.log from today (sep 09), any
>> ideas?
>>
>> Thanks for the support.
>> Juan Khawly
>>
>> On Wednesday, September 7, 2022 at 8:59:22 PM UTC-4 Juan Khawly wrote:
>>
>>> Makes total sense.
>>>
>>> Just added the option to the DaemonProcess and LogLevel info to the
>>> virtual host. I will be monitoring the the logs and report back in a couple
>>> days for reference.
>>>
>>> Appreciate your help.
>>> Juan Khawly
>>>
>>> On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton
>>> wrote:
>>>
>>>>
>>>> On 7 Sep 2022, at 11:43 pm, Juan Khawly <juan...@gmail.com> wrote:
>>>>
>>>> Graham,
>>>>
>>>> Going to make that change, monitor and keep this chat updated with the
>>>> result.
>>>>
>>>> 2 Questions:
>>>>
>>>> 1) The option request-timeout = 60 is included inside the virtual host
>>>> along with the rest of the Daemon code right ?
>>>>
>>>>
>>>> It is an option to be added to the existing WSGIDaemonProcess directive.
>>>>
>>>> 2) *Under no traffic*, do you have any idea of why this problem could
>>>> happen? As I explained, it is usually, but not always, preceded by couple
>>>> of GET Request from a random IP (bot requests) to random urls. My
>>>> assumption was Slow DDOS and this is why I enabled modreqtimeout, mod
>>>> security and mod qos. But at this point I'm clueless of how to diagnose.
>>>>
>>>>
>>>> No idea. If it was truly a slow DDOS attack the request wouldn't
>>>> actually show in the access logs because Apache only logs requests on
>>>> completion. So am not sure one could say is related to those other
>>>> requests. I would say it is more likely that over time a trickle of
>>>> requests come in to your application as normal which block and slowly use
>>>> up capacity. Hopefully the stack trace created when get a forced restart
>>>> due to request timeout will show where. Just keep in mind that since the
>>>> request timeout will cause auto recovery you may not notice it occurred,
>>>> so
>>>> you will need to periodically check Apache error logs yourself. Make sure
>>>> that have info LogLevel for the virtual host so get more useful
>>>> information
>>>> out of mod_wsgi.
>>>>
>>>> Thanks
>>>> Juan Khawly
>>>>
>>>> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton
>>>> wrote:
>>>>
>>>>> Sorry, seems I didn't see your update.
>>>>>
>>>>> Add an option:
>>>>>
>>>>> request-timeout=60
>>>>>
>>>>> to the WSGIDaemonProcess.
>>>>>
>>>>> Set the value (in seconds) greater than you would expect your HTTP
>>>>> requests to normally run.
>>>>>
>>>>> What will happen is that when the average running time for all
>>>>> possible concurrent requests exceeds that timeout value, the daemon
>>>>> process
>>>>> will be forcibly restarted. This will have the effect of unblocking the
>>>>> process and a new one will be started in its place. So acts as a fail
>>>>> safe
>>>>> to ensure your application keeps running.
>>>>>
>>>>> What this will also do is attempt to dump out Python stack traces for
>>>>> what all the request handler threads were doing when the process is
>>>>> restarted. This will hopefully allow you to work out why your request
>>>>> handlers are getting stuck, be it they are getting stuck on a lock, or
>>>>> waiting on a backend service.
>>>>>
>>>>> In short, your request handlers are getting stuck and not completing.
>>>>> Over time these are building up and the thread pool for handling requests
>>>>> is exhausted and so the process stops handling requests.
>>>>>
>>>>> Graham
>>>>>
>>>>> On 7 Sep 2022, at 5:44 am, Juan Khawly <juan...@gmail.com> wrote:
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote:
>>>>>
>>>>>> Hello Graham,
>>>>>>
>>>>>> I'm going to try to address your questions:
>>>>>>
>>>>>> *Inside my Virtual Host*
>>>>>>
>>>>>> Alias /static /data/home/user/project/frontend/build/static
>>>>>> <Directory /data/home/user/project/frontend/build/static>
>>>>>> Require all granted
>>>>>> </Directory>
>>>>>>
>>>>>> <Directory /data/home/user/project/my_project>
>>>>>> <Files wsgi.py>
>>>>>> Require all granted
>>>>>> </Files>
>>>>>> </Directory>
>>>>>>
>>>>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py
>>>>>> WSGIDaemonProcess my_project
>>>>>> python-path=/data/home/user/project
>>>>>> python-home=/data/home/user/environment/venv
>>>>>> WSGIProcessGroup my_project
>>>>>>
>>>>>>
>>>>>> *Inside apache2.conf*
>>>>>>
>>>>>> WSGIApplicationGroup %{GLOBAL}
>>>>>>
>>>>>> *On the apache/error.log*
>>>>>> When I get the 503 on the access.log, these are the types of errors
>>>>>> seen on the error.log
>>>>>>
>>>>>> *One type of error*
>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>>>> response headers from daemon process 'my_project':
>>>>>> /data/home/project/my_project/my_project/wsgi.py
>>>>>>
>>>>>> *Another type of error*
>>>>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid
>>>>>> 140518595991296] (11)Resource temporarily unavailable: [client
>>>>>> 172.31.17.102:31880] mod_wsgi (pid=3267): Unable to connect to WSGI
>>>>>> daemon process ' my_project ' on '/var/run/apache2/wsgi.2385.1.1.sock'
>>>>>> after multiple attempts as listener backlog limit was exceeded or the
>>>>>> socket does not exist.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Juan Khawly
>>>>>>
>>>>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton
>>>>>> wrote:
>>>>>>
>>>>>>> Would need to see the mod_wsgi configuration you are using to
>>>>>>> configure the WSGI application, including how WSGIDaemonProcess is
>>>>>>> configured and whether you are using WSGIApplicationGroup. Also, what
>>>>>>> errors are in the Apache error log when the 503 errors occur.
>>>>>>>
>>>>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I've been running into this problem for a while.
>>>>>>>
>>>>>>> *CONTEXT *
>>>>>>>
>>>>>>> I have an application developed in python (3.10), django 4.0.3,
>>>>>>> using mod_wsgi and apache. The application is in a DEV environment and
>>>>>>> hosted in AWS EC2. Currently, it does not receive traffic at all.
>>>>>>>
>>>>>>> *Installation of Mod WSGI*
>>>>>>> apt-get install -y apache2-dev
>>>>>>>
>>>>>>> *Setup out of the VENV*
>>>>>>> mod_wsgi-express install-module
>>>>>>>
>>>>>>> editing: /etc/apache2/mods-available/wsgi.load
>>>>>>>
>>>>>>> LoadModule wsgi_module "/usr/lib/apache2/modules/
>>>>>>> mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
>>>>>>> WSGIPythonHome "/data/home/user/environment/venv"
>>>>>>>
>>>>>>> *Module Enabled*
>>>>>>> a2enmod wsgi
>>>>>>>
>>>>>>> *PROBLEM*
>>>>>>>
>>>>>>> The application works perfect most of the time. Couple of times a
>>>>>>> week, without traffic the apache server goes down into 503. Usually it
>>>>>>> is
>>>>>>> preceded by a random request but it does not always happen that way. I
>>>>>>> am
>>>>>>> assuming that is Slow DDOS but I want to make sure it is not miss
>>>>>>> configuration of the WSGI.
>>>>>>>
>>>>>>> access.log example
>>>>>>>
>>>>>>> <access.PNG>
>>>>>>>
>>>>>>> error.log example
>>>>>>> I masked the internal routes
>>>>>>>
>>>>>>> *This is one of the errors:*
>>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>>>>> response headers from daemon process 'XXXXX':
>>>>>>> /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py
>>>>>>>
>>>>>>> *Another type of error:*
>>>>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>>>>> response headers from daemon process 'XXXXXXX':
>>>>>>> /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py
>>>>>>>
>>>>>>> *SOLUTION*
>>>>>>>
>>>>>>> If I restart the server, all works again until next failure.
>>>>>>>
>>>>>>> I've enabled the following modules, in case it is SlowDDOS
>>>>>>> modreqtimeout
>>>>>>> libapache2-mod-qos
>>>>>>> libapache2-mod-security2.
>>>>>>>
>>>>>>> Any recommendation?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Juan Khawly
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "modwsgi" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to modwsgi+u...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com
>>>>>>>
>>>>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>> <access.PNG>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "modwsgi" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to modwsgi+u...@googlegroups.com.
>>>>>
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>>
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to modwsgi+u...@googlegroups.com.
>>>>
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>>
>>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to modwsgi+u...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/4c08f3cf-54f6-43a3-809d-a463723c274cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> <error.log>
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to modwsgi+u...@googlegroups.com.
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/modwsgi/a69d70f4-faac-4515-b991-90590efd7b95n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to modwsgi+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/33b4cf84-410f-48d8-a062-4d1ef5e92a50n%40googlegroups.com.
