I'll watch. Thank you! On Tuesday, August 11, 2020 at 11:39:27 AM UTC+3, Graham Dumpleton wrote: > > Did you watch the videos? That is covered in some of them. > > On 11 Aug 2020, at 5:57 pm, Paul Royik <[email protected] <javascript:>> > wrote: > > Can you at least tell me how apache settings coordinate with mod_wsgi > settings? > > I set number of threads, processes in Apache and in mod_wsgi. How do they > work together? > > In mod_wsgi processes=5 threads=12. Does it mean 60 concurrent requests? > > On Tuesday, August 11, 2020 at 10:31:56 AM UTC+3, Graham Dumpleton wrote: >> >> The problem was you had other odd requirements like having non thread >> safe code. >> >> All I can do is suggest you watch these videos. >> >> https://www.youtube.com/watch?v=CPz0s1CQsTE&t=4s >> https://www.youtube.com/watch?v=SGleKfigMsk&t=2s >> https://www.youtube.com/watch?v=k6Erh7oHvns&t=1s >> >> There is no simple answer as one needs to know how your application works >> and have access to performance metrics from an APM service in order to tune >> the server. >> >> Graham >> >> On 11 Aug 2020, at 5:27 pm, Paul Royik <[email protected]> wrote: >> >> One more question. >> >> What is a good configuration if I have 500 concurrent requests and there >> are no long-running tasks? >> >> On Tuesday, August 11, 2020 at 9:52:39 AM UTC+3, Paul Royik wrote: >>> >>> Thank you for your help. >>> >>> On Tuesday, August 11, 2020 at 8:30:27 AM UTC+3, Graham Dumpleton wrote: >>>> >>>> If it is a minor quick running script that does something simple it >>>> should be okay. It is just having long running processes would be more >>>> concerned about. >>>> >>>> On 11 Aug 2020, at 3:18 pm, Paul Royik <[email protected]> wrote: >>>> >>>> You are absolutely correct. Need to change the architecture. >>>> One more question. I also use subprocess.check_output from django. Is >>>> it also bad idea? I'm trying to run a script (non-python) and get it >>>> output. >>>> >>>> On Tuesday, August 11, 2020 at 1:55:51 AM UTC+3, Graham Dumpleton wrote: >>>>> >>>>> Personally I would be concerned about the architecture you are using >>>>> if you have long running tasks like you describe. It is not usually a >>>>> good >>>>> idea to use 'multiprocessing.Process' to create sub processes directly >>>>> from >>>>> a web server process to perform work. A better architecture would be to >>>>> off >>>>> load the work into a queue using something like Celery and have the >>>>> separate job processing system pull the jobs from the queue and process >>>>> them. You would also be better off to model the interaction from the >>>>> front >>>>> end as queueing the job and immediately responding with an >>>>> acknowledgement >>>>> to say is queued. The front end can then start polling periodically to >>>>> see >>>>> if the job has finished, and when it has it would get the response back. >>>>> The front end can then display the data or save it locally as needed. >>>>> >>>>> This model avoids the problem of requests being parked doing nothing >>>>> for a long time, which with your server configuration is going to be >>>>> hugely >>>>> expensive on memory and not scale very well because of limitations of >>>>> using >>>>> WSGI process/threading model. You might even consider not using a WSGI >>>>> application at all. Instead, use an async web application paired with >>>>> Celery for execution of the jobs. Using an async web application means >>>>> you >>>>> can handle as many parked requests as you want and they can quite happily >>>>> sit there waiting for Celery to finish the job and don't need to use >>>>> polling. Only thing am not sure about in that is what async clients there >>>>> are for Celery. >>>>> >>>>> Graham >>>>> >>>>> On 10 Aug 2020, at 9:09 pm, Paul Royik <[email protected]> wrote: >>>>> >>>>> My django app makes heavy calculations which can be infinite. >>>>> That's why, when user enters the site, i.e. makes a request, heavy >>>>> calculations are wrapped into multiprocessing.Process which runs at most >>>>> 7 >>>>> seconds. >>>>> I can't use threads, because third-party packages are not thread-safe. >>>>> >>>>> So, I have around 30 concurrent requests per second. If each request >>>>> can take up to 7 seconds, then it is 30*7=210 concurrent requests in the >>>>> worst case. >>>>> Each of these concurrent requests opens multiprocessing.Process, >>>>> which gives (I guess) 210*2=420 (close to 500) concurrent requests in the >>>>> worst case. >>>>> That' how I got 500 requests. Possibly, my calculations are incorrect. >>>>> >>>>> Average page load time (average response times) is 10 seconds. >>>>> >>>>> I use MPM worker. >>>>> >>>>> I set WSGIProcessGroup >>>>> >>>>> StartServers 100 >>>>> ServerLimit 500 >>>>> >>>>> ThreadsPerChild 1 >>>>> ThreadLimit 1 >>>>> >>>>> MaxRequestWorkers 500 >>>>> MaxConnectionsPerChild 10000 >>>>> >>>>> WSGIApplicationGroup %{GLOBAL} >>>>> WSGIDaemonProcess django_app processes=75 threads=1 python-path='...' >>>>> maximum-requests=10000 request-timeout=20 >>>>> WSGIProcessGroup django_app >>>>> >>>>> WSGIRestrictEmbedded On >>>>> WSGILazyInitialization On >>>>> >>>>> >>>>> >>>>> On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton wrote: >>>>>> >>>>>> What sort of application are you running? >>>>>> >>>>>> What is your average response times? >>>>>> >>>>>> Do you have long running requests, if yes, how long? >>>>>> >>>>>> What Apache MPM are you actually using? >>>>>> >>>>>> My initial impression is that is a quite poor configuration which is >>>>>> only going to chew up huge amounts of memory for no good reason, but I >>>>>> don't know your application requirements. >>>>>> >>>>>> Also, are you even setting WSGIProcessGroup? If it isn't set it >>>>>> makes the whole daemon process configuration moot as it isn't even being >>>>>> used. >>>>>> >>>>>> On 10 Aug 2020, at 7:24 pm, Paul Royik <[email protected]> wrote: >>>>>> >>>>>> StartServers 50 >>>>>> ServerLimit 200 >>>>>> >>>>>> ThreadsPerChild 1 >>>>>> ThreadLimit 1 >>>>>> >>>>>> MaxRequestWorkers 200 >>>>>> MaxConnectionsPerChild 10000 >>>>>> >>>>>> WSGIApplicationGroup %{GLOBAL} >>>>>> WSGIDaemonProcess process processes=75 threads=1 >>>>>> >>>>>> >>>>>> >>>>>> Is it enough? Or can it handle only 75 concurrent requests? I don't know >>>>>> how to synchronize apache and mod_wsgi settings. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "modwsgi" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > >
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/92149949-ce00-4868-a782-7f2b086cbba7o%40googlegroups.com.
