Regarding your question on average response time and longest resposnse.
Average response times for most of the queries is a maximum of 600 ms.
Longest response times is about 30 seconds and some apis that integrate
with 3rd party apps take 5 seconds average. So we overall have some long
running calls > 1 sec due to the transactional nature of the application.
Thanks,
Lalith
On Tuesday, December 14, 2021 at 9:41:45 AM UTC-6 Lalith Maddali wrote:
> Graham, I Really appreciate the response. I will work with my team to test
> these changes and get back to you!
>
> Thanks,
> Lalith
>
> On Tuesday, December 14, 2021 at 12:49:40 AM UTC-6 Graham Dumpleton wrote:
>
>> Will possible look and comment more later, but the very first thing you
>> want to do is change:
>>
>> WSGIProcessGroup yyy.bbb.com
>> WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py
>>
>> to:
>>
>> WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py process-group=
>> yyy.bbb.com application-group=%{GLOBAL}
>>
>> The most important bit is setting the application group to %{GLOBAL}.
>> This avoids problems with some third party modules which aren't designed to
>> work in Python sub interpreters and which can cause deadlocks with threads
>> causing them to hang and slowly you run out of capacity.
>>
>> Other comments are you should not need:
>>
>> SetHandler wsgi-script
>>
>> and it is recommend to set outside of the VirtualHost:
>>
>> WSGIRestrictEmbedded On
>>
>> This last one disables Python in main Apache child worker processes,
>> saving on memory and startup times for those. Because you are using daemon
>> mode you don't need embedded mode enabled.
>>
>> Using maximum-requests is actually usually not a good idea because when
>> site under load then it causes process to restart too frequently, making
>> things worse.
>>
>> You should instead look at restart-interval option for periodic restarts.
>>
>> Also perhaps want to look at request-timeout option to force process
>> restarts when requests run too long and/or hang.
>>
>> What is the average response time, plus longest response times you expect?
>>
>> If you haven't watch the talk:
>>
>> https://www.youtube.com/watch?v=H6Q3l11fjU0
>>
>> Graham
>>
>> On 14 Dec 2021, at 5:41 pm, Lalith Maddali <[email protected]> wrote:
>>
>> Thank you for the prompt reply Graham.
>>
>> Box capacity: 4 CPU, 16 GB RAM, 14 GB Swap space
>>
>> Python version: 2.7
>> Django version: 1.8.4 final
>> Mod_wsgi version: 4.7.1
>>
>> Here is Apache configuration. I have masked the internal names with xxx,
>> yyy, bbb but it should be consistent across the whole file.
>>
>> WSGISocketPrefix /var/run/
>>
>> WSGIPassAuthorization On
>>
>> <Directory "/var/www/html/xxx/yyy">
>>
>> #Options FollowSymLinks
>>
>> #AllowOverride All
>>
>> Options Indexes FollowSymLinks
>>
>> AllowOverride All
>>
>> </Directory>
>>
>> <VirtualHost *:443>
>>
>> ServerName yyy.bbb.com
>>
>> ServerAlias yyy.staging.bbb.com
>>
>> DocumentRoot "/var/www/html/xxx/yyy"
>>
>> WSGIDaemonProcess yyy.bbb.com \
>>
>> user=apache group=apache \
>>
>> processes=6 threads=20 \
>>
>> display-name='%{GROUP}' \
>>
>> maximum-requests=1800 graceful-timeout=720 socket-timeout=300 \
>>
>> python-path=/var/www/html/xxx/:/var/www/html/xxx/ccc
>>
>> WSGIProcessGroup yyy.bbb.com
>>
>> WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py
>>
>> SetHandler wsgi-script SSLEngine on
>>
>> </VirtualHost>
>>
>> Please let me know if there is anything else I can provide. Thank you for
>> your help!
>>
>> Thanks,
>> Lalith
>>
>> On Monday, December 13, 2021 at 8:50:55 PM UTC-6 Graham Dumpleton wrote:
>>
>>> Can you supply the mod_wsgi configuration you are using in Apache
>>> configuration file.
>>>
>>> I would need to see mod_wsgi directives such as WSGIDaemonProcess,
>>> WSGIProcessGroup, WSGIApplicationGroup, WSGIScriptAlias and how they are
>>> used in the context of Apache configuration.
>>>
>>> Graham
>>>
>>> On 14 Dec 2021, at 1:47 pm, Lalith Maddali <[email protected]> wrote:
>>>
>>> Hi All,
>>>
>>> Occassionally users reported that UI was very slow. When we dig
>>> deeper we found that some requests never got a response back, rather after
>>> loadbalancer timed out it returned 504 status code.
>>>
>>> Most API calls are high in request size (due to users saving a lot of
>>> text data back to server). I saw some blogs saying that large request sizes
>>> can bloat the memory consumed by django process. We were definitely hitting
>>> peak memory due to this.
>>>
>>> We introduced maximum-request setting to kill process once it reaches
>>> this limit. It was fine until we hit this weird issue of timeouts for
>>> seemingly simple API calls.
>>>
>>> Digging further, we found that in Apache logs there are 403 status
>>> codes, it happens right during the time Apache restarts the processes as
>>> maximum-request setting is reached. We could see some logs of django
>>> process start logs around the same time.
>>>
>>> Question is why do we have 403? In our other production servers this was
>>> never an issue, only on this one application, Apache keeps the requests
>>> dangling and later 403 and not sending to the active processes. This is
>>> requiring us to periodically do apache graceful restarts. This is not
>>> elegant solution, but any insight into what to improve would be great.
>>>
>>> We also have New Relic installed as middleware. We found that a process
>>> restart takes more than 20 seconds. Is this the root cause? How do we deal
>>> with improving process restart?
>>>
>>> Thanks,
>>> Lalith
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/4cb4025a-7dc6-4993-8dd6-78f7492a0607n%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/4cb4025a-7dc6-4993-8dd6-78f7492a0607n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/a76046f7-8f30-4756-9a97-6bcc5fca2024n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/a76046f7-8f30-4756-9a97-6bcc5fca2024n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/16761a72-0ab4-4396-bd70-fe1aceaf663cn%40googlegroups.com.
