While researching things, also read: http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html <http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html>
It can be advantageous to run multiple daemon process groups and use Location/WSGIProcessGroup to direct subsets of the URL namespace to a different process group and application instance. By doing this you can better tune each respective daemon process groups for the type of traffic it receives, with a candidate being to segment out long running requests to their own process group so can set high request timeouts, with lower request timeout in core process group handling everything else. Graham > On 15 Dec 2021, at 4:00 am, Lalith Maddali <[email protected]> wrote: > > Regarding your question on average response time and longest resposnse. > > Average response times for most of the queries is a maximum of 600 ms. > > Longest response times is about 30 seconds and some apis that integrate with > 3rd party apps take 5 seconds average. So we overall have some long running > calls > 1 sec due to the transactional nature of the application. > > Thanks, > Lalith > > On Tuesday, December 14, 2021 at 9:41:45 AM UTC-6 Lalith Maddali wrote: > Graham, I Really appreciate the response. I will work with my team to test > these changes and get back to you! > > Thanks, > Lalith > > On Tuesday, December 14, 2021 at 12:49:40 AM UTC-6 Graham Dumpleton wrote: > Will possible look and comment more later, but the very first thing you want > to do is change: > > WSGIProcessGroup yyy.bbb.com <http://yyy.bbb.com/> > WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py > > to: > > WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py process-group=yyy.bbb.com > <http://yyy.bbb.com/> application-group=%{GLOBAL} > > The most important bit is setting the application group to %{GLOBAL}. This > avoids problems with some third party modules which aren't designed to work > in Python sub interpreters and which can cause deadlocks with threads causing > them to hang and slowly you run out of capacity. > > Other comments are you should not need: > > SetHandler wsgi-script > > and it is recommend to set outside of the VirtualHost: > > WSGIRestrictEmbedded On > > This last one disables Python in main Apache child worker processes, saving > on memory and startup times for those. Because you are using daemon mode you > don't need embedded mode enabled. > > Using maximum-requests is actually usually not a good idea because when site > under load then it causes process to restart too frequently, making things > worse. > > You should instead look at restart-interval option for periodic restarts. > > Also perhaps want to look at request-timeout option to force process restarts > when requests run too long and/or hang. > > What is the average response time, plus longest response times you expect? > > If you haven't watch the talk: > > https://www.youtube.com/watch?v=H6Q3l11fjU0 > <https://www.youtube.com/watch?v=H6Q3l11fjU0> > > Graham > > >> On 14 Dec 2021, at 5:41 pm, Lalith Maddali <[email protected] <>> wrote: >> > >> Thank you for the prompt reply Graham. >> >> Box capacity: 4 CPU, 16 GB RAM, 14 GB Swap space >> >> Python version: 2.7 >> Django version: 1.8.4 final >> Mod_wsgi version: 4.7.1 >> >> Here is Apache configuration. I have masked the internal names with xxx, >> yyy, bbb but it should be consistent across the whole file. >> >> WSGISocketPrefix /var/run/ >> >> WSGIPassAuthorization On >> >> <Directory "/var/www/html/xxx/yyy"> >> >> #Options FollowSymLinks >> >> #AllowOverride All >> >> Options Indexes FollowSymLinks >> >> AllowOverride All >> >> </Directory> >> >> <VirtualHost *:443> >> >> ServerName yyy.bbb.com <http://yyy.bbb.com/> >> >> ServerAlias yyy.staging.bbb.com <http://yyy.staging.bbb.com/> >> >> DocumentRoot "/var/www/html/xxx/yyy" >> >> WSGIDaemonProcess yyy.bbb.com <http://yyy.bbb.com/> \ >> >> user=apache group=apache \ >> >> processes=6 threads=20 \ >> >> display-name='%{GROUP}' \ >> >> maximum-requests=1800 graceful-timeout=720 socket-timeout=300 \ >> >> >> python-path=/var/www/html/xxx/:/var/www/html/xxx/ccc >> >> WSGIProcessGroup yyy.bbb.com <http://yyy.bbb.com/> >> >> WSGIScriptAlias / /var/www/html/xxx/yyy/wsgi.py >> >> SetHandler wsgi-script SSLEngine on >> >> </VirtualHost> >> >> Please let me know if there is anything else I can provide. Thank you for >> your help! >> >> Thanks, >> Lalith >> >> >> On Monday, December 13, 2021 at 8:50:55 PM UTC-6 Graham Dumpleton wrote: >> Can you supply the mod_wsgi configuration you are using in Apache >> configuration file. >> >> I would need to see mod_wsgi directives such as WSGIDaemonProcess, >> WSGIProcessGroup, WSGIApplicationGroup, WSGIScriptAlias and how they are >> used in the context of Apache configuration. >> >> Graham >> >> >>> On 14 Dec 2021, at 1:47 pm, Lalith Maddali <[email protected] <>> wrote: >>> >> >>> Hi All, >>> >>> Occassionally users reported that UI was very slow. When we dig >>> deeper we found that some requests never got a response back, rather after >>> loadbalancer timed out it returned 504 status code. >>> >>> Most API calls are high in request size (due to users saving a lot of text >>> data back to server). I saw some blogs saying that large request sizes can >>> bloat the memory consumed by django process. We were definitely hitting >>> peak memory due to this. >>> >>> We introduced maximum-request setting to kill process once it reaches this >>> limit. It was fine until we hit this weird issue of timeouts for seemingly >>> simple API calls. >>> >>> Digging further, we found that in Apache logs there are 403 status codes, >>> it happens right during the time Apache restarts the processes as >>> maximum-request setting is reached. We could see some logs of django >>> process start logs around the same time. >>> >>> Question is why do we have 403? In our other production servers this was >>> never an issue, only on this one application, Apache keeps the requests >>> dangling and later 403 and not sending to the active processes. This is >>> requiring us to periodically do apache graceful restarts. This is not >>> elegant solution, but any insight into what to improve would be great. >>> >>> We also have New Relic installed as middleware. We found that a process >>> restart takes more than 20 seconds. Is this the root cause? How do we deal >>> with improving process restart? >>> >>> Thanks, >>> Lalith >>> >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] <>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/4cb4025a-7dc6-4993-8dd6-78f7492a0607n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/4cb4025a-7dc6-4993-8dd6-78f7492a0607n%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <>. > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/a76046f7-8f30-4756-9a97-6bcc5fca2024n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/a76046f7-8f30-4756-9a97-6bcc5fca2024n%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/16761a72-0ab4-4396-bd70-fe1aceaf663cn%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/16761a72-0ab4-4396-bd70-fe1aceaf663cn%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/9ACDE112-4D85-4382-B0AC-60D0D53C5CC7%40gmail.com.
