One more question. What is a good configuration if I have 500 concurrent requests and there are no long-running tasks?
On Tuesday, August 11, 2020 at 9:52:39 AM UTC+3, Paul Royik wrote: > > Thank you for your help. > > On Tuesday, August 11, 2020 at 8:30:27 AM UTC+3, Graham Dumpleton wrote: >> >> If it is a minor quick running script that does something simple it >> should be okay. It is just having long running processes would be more >> concerned about. >> >> On 11 Aug 2020, at 3:18 pm, Paul Royik <[email protected]> wrote: >> >> You are absolutely correct. Need to change the architecture. >> One more question. I also use subprocess.check_output from django. Is it >> also bad idea? I'm trying to run a script (non-python) and get it output. >> >> On Tuesday, August 11, 2020 at 1:55:51 AM UTC+3, Graham Dumpleton wrote: >>> >>> Personally I would be concerned about the architecture you are using if >>> you have long running tasks like you describe. It is not usually a good >>> idea to use 'multiprocessing.Process' to create sub processes directly from >>> a web server process to perform work. A better architecture would be to off >>> load the work into a queue using something like Celery and have the >>> separate job processing system pull the jobs from the queue and process >>> them. You would also be better off to model the interaction from the front >>> end as queueing the job and immediately responding with an acknowledgement >>> to say is queued. The front end can then start polling periodically to see >>> if the job has finished, and when it has it would get the response back. >>> The front end can then display the data or save it locally as needed. >>> >>> This model avoids the problem of requests being parked doing nothing for >>> a long time, which with your server configuration is going to be hugely >>> expensive on memory and not scale very well because of limitations of using >>> WSGI process/threading model. You might even consider not using a WSGI >>> application at all. Instead, use an async web application paired with >>> Celery for execution of the jobs. Using an async web application means you >>> can handle as many parked requests as you want and they can quite happily >>> sit there waiting for Celery to finish the job and don't need to use >>> polling. Only thing am not sure about in that is what async clients there >>> are for Celery. >>> >>> Graham >>> >>> On 10 Aug 2020, at 9:09 pm, Paul Royik <[email protected]> wrote: >>> >>> My django app makes heavy calculations which can be infinite. >>> That's why, when user enters the site, i.e. makes a request, heavy >>> calculations are wrapped into multiprocessing.Process which runs at most 7 >>> seconds. >>> I can't use threads, because third-party packages are not thread-safe. >>> >>> So, I have around 30 concurrent requests per second. If each request can >>> take up to 7 seconds, then it is 30*7=210 concurrent requests in the worst >>> case. >>> Each of these concurrent requests opens multiprocessing.Process, which >>> gives (I guess) 210*2=420 (close to 500) concurrent requests in the worst >>> case. >>> That' how I got 500 requests. Possibly, my calculations are incorrect. >>> >>> Average page load time (average response times) is 10 seconds. >>> >>> I use MPM worker. >>> >>> I set WSGIProcessGroup >>> >>> StartServers 100 >>> ServerLimit 500 >>> >>> ThreadsPerChild 1 >>> ThreadLimit 1 >>> >>> MaxRequestWorkers 500 >>> MaxConnectionsPerChild 10000 >>> >>> WSGIApplicationGroup %{GLOBAL} >>> WSGIDaemonProcess django_app processes=75 threads=1 python-path='...' >>> maximum-requests=10000 request-timeout=20 >>> WSGIProcessGroup django_app >>> >>> WSGIRestrictEmbedded On >>> WSGILazyInitialization On >>> >>> >>> >>> On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton wrote: >>>> >>>> What sort of application are you running? >>>> >>>> What is your average response times? >>>> >>>> Do you have long running requests, if yes, how long? >>>> >>>> What Apache MPM are you actually using? >>>> >>>> My initial impression is that is a quite poor configuration which is >>>> only going to chew up huge amounts of memory for no good reason, but I >>>> don't know your application requirements. >>>> >>>> Also, are you even setting WSGIProcessGroup? If it isn't set it makes >>>> the whole daemon process configuration moot as it isn't even being used. >>>> >>>> On 10 Aug 2020, at 7:24 pm, Paul Royik <[email protected]> wrote: >>>> >>>> StartServers 50 >>>> ServerLimit 200 >>>> >>>> ThreadsPerChild 1 >>>> ThreadLimit 1 >>>> >>>> MaxRequestWorkers 200 >>>> MaxConnectionsPerChild 10000 >>>> >>>> WSGIApplicationGroup %{GLOBAL} >>>> WSGIDaemonProcess process processes=75 threads=1 >>>> >>>> >>>> >>>> Is it enough? Or can it handle only 75 concurrent requests? I don't know >>>> how to synchronize apache and mod_wsgi settings. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com.
