Graham, I Really appreciate the response. I will work with my team to test
these changes and get back to you!
Thanks,
Lalith
On Tuesday, December 14, 2021 at 12:49:40 AM UTC-6 Graham Dumpleton wrote:
> Will possible look and comment more later, but the very first thing you
> want to do is change:
>
> WSGIProcessGroup yyy.bbb.com
> WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py
>
> to:
>
> WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py process-group=
> yyy.bbb.com application-group=%{GLOBAL}
>
> The most important bit is setting the application group to %{GLOBAL}. This
> avoids problems with some third party modules which aren't designed to work
> in Python sub interpreters and which can cause deadlocks with threads
> causing them to hang and slowly you run out of capacity.
>
> Other comments are you should not need:
>
> SetHandler wsgi-script
>
> and it is recommend to set outside of the VirtualHost:
>
> WSGIRestrictEmbedded On
>
> This last one disables Python in main Apache child worker processes,
> saving on memory and startup times for those. Because you are using daemon
> mode you don't need embedded mode enabled.
>
> Using maximum-requests is actually usually not a good idea because when
> site under load then it causes process to restart too frequently, making
> things worse.
>
> You should instead look at restart-interval option for periodic restarts.
>
> Also perhaps want to look at request-timeout option to force process
> restarts when requests run too long and/or hang.
>
> What is the average response time, plus longest response times you expect?
>
> If you haven't watch the talk:
>
> https://www.youtube.com/watch?v=H6Q3l11fjU0
>
> Graham
>
> On 14 Dec 2021, at 5:41 pm, Lalith Maddali <[email protected]> wrote:
>
> Thank you for the prompt reply Graham.
>
> Box capacity: 4 CPU, 16 GB RAM, 14 GB Swap space
>
> Python version: 2.7
> Django version: 1.8.4 final
> Mod_wsgi version: 4.7.1
>
> Here is Apache configuration. I have masked the internal names with xxx,
> yyy, bbb but it should be consistent across the whole file.
>
> WSGISocketPrefix /var/run/
>
> WSGIPassAuthorization On
>
> <Directory "/var/www/html/xxx/yyy">
>
> #Options FollowSymLinks
>
> #AllowOverride All
>
> Options Indexes FollowSymLinks
>
> AllowOverride All
>
> </Directory>
>
> <VirtualHost *:443>
>
> ServerName yyy.bbb.com
>
> ServerAlias yyy.staging.bbb.com
>
> DocumentRoot "/var/www/html/xxx/yyy"
>
> WSGIDaemonProcess yyy.bbb.com \
>
> user=apache group=apache \
>
> processes=6 threads=20 \
>
> display-name='%{GROUP}' \
>
> maximum-requests=1800 graceful-timeout=720 socket-timeout=300 \
>
> python-path=/var/www/html/xxx/:/var/www/html/xxx/ccc
>
> WSGIProcessGroup yyy.bbb.com
>
> WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py
>
> SetHandler wsgi-script SSLEngine on
>
> </VirtualHost>
>
> Please let me know if there is anything else I can provide. Thank you for
> your help!
>
> Thanks,
> Lalith
>
> On Monday, December 13, 2021 at 8:50:55 PM UTC-6 Graham Dumpleton wrote:
>
>> Can you supply the mod_wsgi configuration you are using in Apache
>> configuration file.
>>
>> I would need to see mod_wsgi directives such as WSGIDaemonProcess,
>> WSGIProcessGroup, WSGIApplicationGroup, WSGIScriptAlias and how they are
>> used in the context of Apache configuration.
>>
>> Graham
>>
>> On 14 Dec 2021, at 1:47 pm, Lalith Maddali <[email protected]> wrote:
>>
>> Hi All,
>>
>> Occassionally users reported that UI was very slow. When we dig
>> deeper we found that some requests never got a response back, rather after
>> loadbalancer timed out it returned 504 status code.
>>
>> Most API calls are high in request size (due to users saving a lot of
>> text data back to server). I saw some blogs saying that large request sizes
>> can bloat the memory consumed by django process. We were definitely hitting
>> peak memory due to this.
>>
>> We introduced maximum-request setting to kill process once it reaches
>> this limit. It was fine until we hit this weird issue of timeouts for
>> seemingly simple API calls.
>>
>> Digging further, we found that in Apache logs there are 403 status codes,
>> it happens right during the time Apache restarts the processes as
>> maximum-request setting is reached. We could see some logs of django
>> process start logs around the same time.
>>
>> Question is why do we have 403? In our other production servers this was
>> never an issue, only on this one application, Apache keeps the requests
>> dangling and later 403 and not sending to the active processes. This is
>> requiring us to periodically do apache graceful restarts. This is not
>> elegant solution, but any insight into what to improve would be great.
>>
>> We also have New Relic installed as middleware. We found that a process
>> restart takes more than 20 seconds. Is this the root cause? How do we deal
>> with improving process restart?
>>
>> Thanks,
>> Lalith
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/4cb4025a-7dc6-4993-8dd6-78f7492a0607n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/4cb4025a-7dc6-4993-8dd6-78f7492a0607n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
>
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/a76046f7-8f30-4756-9a97-6bcc5fca2024n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/modwsgi/a76046f7-8f30-4756-9a97-6bcc5fca2024n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/2b59c379-93ee-4ac2-95e2-c539e8b390b3n%40googlegroups.com.
