Makes total sense.
Just added the option to the DaemonProcess and LogLevel info to the virtual
host. I will be monitoring the the logs and report back in a couple days
for reference.
Appreciate your help.
Juan Khawly
On Wednesday, September 7, 2022 at 6:30:17 PM UTC-4 Graham Dumpleton wrote:
>
> On 7 Sep 2022, at 11:43 pm, Juan Khawly <juan...@gmail.com> wrote:
>
> Graham,
>
> Going to make that change, monitor and keep this chat updated with the
> result.
>
> 2 Questions:
>
> 1) The option request-timeout = 60 is included inside the virtual host
> along with the rest of the Daemon code right ?
>
>
> It is an option to be added to the existing WSGIDaemonProcess directive.
>
> 2) *Under no traffic*, do you have any idea of why this problem could
> happen? As I explained, it is usually, but not always, preceded by couple
> of GET Request from a random IP (bot requests) to random urls. My
> assumption was Slow DDOS and this is why I enabled modreqtimeout, mod
> security and mod qos. But at this point I'm clueless of how to diagnose.
>
>
> No idea. If it was truly a slow DDOS attack the request wouldn't actually
> show in the access logs because Apache only logs requests on completion. So
> am not sure one could say is related to those other requests. I would say
> it is more likely that over time a trickle of requests come in to your
> application as normal which block and slowly use up capacity. Hopefully the
> stack trace created when get a forced restart due to request timeout will
> show where. Just keep in mind that since the request timeout will cause
> auto recovery you may not notice it occurred, so you will need to
> periodically check Apache error logs yourself. Make sure that have info
> LogLevel for the virtual host so get more useful information out of
> mod_wsgi.
>
> Thanks
> Juan Khawly
>
> On Tuesday, September 6, 2022 at 6:04:39 PM UTC-4 Graham Dumpleton wrote:
>
>> Sorry, seems I didn't see your update.
>>
>> Add an option:
>>
>> request-timeout=60
>>
>> to the WSGIDaemonProcess.
>>
>> Set the value (in seconds) greater than you would expect your HTTP
>> requests to normally run.
>>
>> What will happen is that when the average running time for all possible
>> concurrent requests exceeds that timeout value, the daemon process will be
>> forcibly restarted. This will have the effect of unblocking the process and
>> a new one will be started in its place. So acts as a fail safe to ensure
>> your application keeps running.
>>
>> What this will also do is attempt to dump out Python stack traces for
>> what all the request handler threads were doing when the process is
>> restarted. This will hopefully allow you to work out why your request
>> handlers are getting stuck, be it they are getting stuck on a lock, or
>> waiting on a backend service.
>>
>> In short, your request handlers are getting stuck and not completing.
>> Over time these are building up and the thread pool for handling requests
>> is exhausted and so the process stops handling requests.
>>
>> Graham
>>
>> On 7 Sep 2022, at 5:44 am, Juan Khawly <juan...@gmail.com> wrote:
>>
>> Any ideas?
>>
>> Thanks
>>
>> On Friday, September 2, 2022 at 8:47:47 AM UTC-4 Juan Khawly wrote:
>>
>>> Hello Graham,
>>>
>>> I'm going to try to address your questions:
>>>
>>> *Inside my Virtual Host*
>>>
>>> Alias /static /data/home/user/project/frontend/build/static
>>> <Directory /data/home/user/project/frontend/build/static>
>>> Require all granted
>>> </Directory>
>>>
>>> <Directory /data/home/user/project/my_project>
>>> <Files wsgi.py>
>>> Require all granted
>>> </Files>
>>> </Directory>
>>>
>>> WSGIScriptAlias / /data/home/user/project/my_project/wsgi.py
>>> WSGIDaemonProcess my_project
>>> python-path=/data/home/user/project
>>> python-home=/data/home/user/environment/venv
>>> WSGIProcessGroup my_project
>>>
>>>
>>> *Inside apache2.conf*
>>>
>>> WSGIApplicationGroup %{GLOBAL}
>>>
>>> *On the apache/error.log*
>>> When I get the 503 on the access.log, these are the types of errors seen
>>> on the error.log
>>>
>>> *One type of error*
>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>> response headers from daemon process 'my_project':
>>> /data/home/project/my_project/my_project/wsgi.py
>>>
>>> *Another type of error*
>>> [Thu Sep 01 04:27:00.053558 2022] [wsgi:error] [pid 3267:tid
>>> 140518595991296] (11)Resource temporarily unavailable: [client
>>> 172.31.17.102:31880] mod_wsgi (pid=3267): Unable to connect to WSGI
>>> daemon process ' my_project ' on '/var/run/apache2/wsgi.2385.1.1.sock'
>>> after multiple attempts as listener backlog limit was exceeded or the
>>> socket does not exist.
>>>
>>>
>>>
>>> Thanks,
>>> Juan Khawly
>>>
>>> On Thursday, September 1, 2022 at 5:54:43 PM UTC-4 Graham Dumpleton
>>> wrote:
>>>
>>>> Would need to see the mod_wsgi configuration you are using to configure
>>>> the WSGI application, including how WSGIDaemonProcess is configured and
>>>> whether you are using WSGIApplicationGroup. Also, what errors are in the
>>>> Apache error log when the 503 errors occur.
>>>>
>>>> On 2 Sep 2022, at 4:57 am, Juan Khawly <juan...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I've been running into this problem for a while.
>>>>
>>>> *CONTEXT *
>>>>
>>>> I have an application developed in python (3.10), django 4.0.3, using
>>>> mod_wsgi and apache. The application is in a DEV environment and hosted in
>>>> AWS EC2. Currently, it does not receive traffic at all.
>>>>
>>>> *Installation of Mod WSGI*
>>>> apt-get install -y apache2-dev
>>>>
>>>> *Setup out of the VENV*
>>>> mod_wsgi-express install-module
>>>>
>>>> editing: /etc/apache2/mods-available/wsgi.load
>>>>
>>>> LoadModule wsgi_module "/usr/lib/apache2/modules/
>>>> mod_wsgi-py310.cpython-310-x86_64-linux-gnu.so"
>>>> WSGIPythonHome "/data/home/user/environment/venv"
>>>>
>>>> *Module Enabled*
>>>> a2enmod wsgi
>>>>
>>>> *PROBLEM*
>>>>
>>>> The application works perfect most of the time. Couple of times a week,
>>>> without traffic the apache server goes down into 503. Usually it is
>>>> preceded by a random request but it does not always happen that way. I am
>>>> assuming that is Slow DDOS but I want to make sure it is not miss
>>>> configuration of the WSGI.
>>>>
>>>> access.log example
>>>>
>>>> <access.PNG>
>>>>
>>>> error.log example
>>>> I masked the internal routes
>>>>
>>>> *This is one of the errors:*
>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>> response headers from daemon process 'XXXXX':
>>>> /XXX/XXXX/XXXXX/XXXXX/XXXXXX/wsgi.py
>>>>
>>>> *Another type of error:*
>>>> [Thu Sep 01 04:22:21.520772 2022] [wsgi:error] [pid 3267:tid
>>>> 140518453380864] [client 118.126.82.157:37722] Timeout when reading
>>>> response headers from daemon process 'XXXXXXX':
>>>> /XXX/XXXX/XXXXX/XXXXXX/XXXXXXX/wsgi.py
>>>>
>>>> *SOLUTION*
>>>>
>>>> If I restart the server, all works again until next failure.
>>>>
>>>> I've enabled the following modules, in case it is SlowDDOS
>>>> modreqtimeout
>>>> libapache2-mod-qos
>>>> libapache2-mod-security2.
>>>>
>>>> Any recommendation?
>>>>
>>>> Thanks,
>>>> Juan Khawly
>>>>
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to modwsgi+u...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/modwsgi/3cc5285a-9943-4143-9b7f-5fa24e681c70n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>> <access.PNG>
>>>>
>>>>
>>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to modwsgi+u...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/d837af27-86cb-4ef6-877a-40f52418a8d0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to modwsgi+u...@googlegroups.com.
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com
>
> <https://groups.google.com/d/msgid/modwsgi/db12a0a4-0acf-4c90-b8a8-4b749f493f7an%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to modwsgi+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/dd04435d-4293-45d8-a399-20ee61872569n%40googlegroups.com.
