I din't delete anything. This is default Webfaction configuration. On Friday, August 14, 2020 at 6:17:21 AM UTC+3, Graham Dumpleton wrote: > > Is there a reason you discarded the default Apache configuration? > > It is important you do not do that as it contains defaults that keep your > system more secure. By deleting all the default config, you have lessoned > the security of your system. > > Also, when you delete the configuration, you don't end up with the same > defaults for many values. This is because the compiled in defaults are > often not the same as is set in the files. It is therefore going to be far > from clear what configuration your system is even running with at this > point. > > Graham > > On 14 Aug 2020, at 12:09 am, Paul Royik <[email protected] <javascript:>> > wrote: > > That's all I have in httpd.conf: > > ServerRoot "/home/user/webapps/django_app/apache2" > > LoadModule alias_module modules/mod_alias.so > LoadModule authz_core_module modules/mod_authz_core.so > LoadModule dir_module modules/mod_dir.so > LoadModule env_module modules/mod_env.so > LoadModule log_config_module modules/mod_log_config.so > LoadModule mime_module modules/mod_mime.so > LoadModule rewrite_module modules/mod_rewrite.so > LoadModule setenvif_module modules/mod_setenvif.so > LoadModule wsgi_module modules/mod_wsgi.so > LoadModule unixd_module modules/mod_unixd.so > LoadModule deflate_module modules/mod_deflate.so > > RewriteEngine on > RewriteCond %{HTTP_REFERER} motherboard.vice.com [NC] > RewriteRule .* - [F] > > LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" > \"%{User-Agent}i\"" combined > CustomLog /home/user/logs/user/access_django_app.log combined > ErrorLog /home/user/logs/user/error_django_app.log > > Listen 18136 > KeepAlive Off > SetEnvIf X-Forwarded-SSL on HTTPS=1 > > MaxRequestWorkers 200 > MaxConnectionsPerChild 10000 > > WSGIDaemonProcess django_app processes=15 threads=5 > python-path=/home/user/webapps/django_app:/home/user/webapps/django_app/lib/python3.7 > maximum-requests=10000 request-timeout=20 > WSGIProcessGroup django_app > WSGIRestrictEmbedded On > WSGILazyInitialization On > WSGIScriptAlias / /path_to_wsgi.py > > > > > On Thursday, August 13, 2020 at 12:58:54 AM UTC+3, Graham Dumpleton wrote: >> >> Setting of MaxRequestWorker (MaxClients) has connections to other MPM >> settings, so must see those as well. >> >> Also usually would be no need to set MaxConnectionsPerChild either if >> everything else is setup correctly when using mod_wsgi daemon mode. >> >> Since using mod_wsgi daemon mode there isn't a need to recycle Apache >> child worker processes, nor restrict the Apache child workers so they are >> single threaded >> >> Rather than cut and paste piece meal parts of your configuration, I need >> to see the whole mod_wsgi parts of the configuration, plus all the Apache >> MPM settings. >> >> So need to see all the settings for the following, depending on whether >> using worker or event MPM. The preferred these days is the "event" MPM. >> >> <IfModule mpm_worker_module> >> StartServers 3 >> MinSpareThreads 75 >> MaxSpareThreads 250 >> ThreadsPerChild 25 >> MaxRequestWorkers 400 >> MaxConnectionsPerChild 0 >> </IfModule> >> >> <IfModule mpm_event_module> >> StartServers 3 >> MinSpareThreads 75 >> MaxSpareThreads 250 >> ThreadsPerChild 25 >> MaxRequestWorkers 400 >> MaxConnectionsPerChild 0 >> </IfModule> >> >> Also need to see what you have for: >> >> WSGIDaemonProcess >> WSGIProcessGroup >> WSGIScriptAlias >> >> WSGIRestrictEmbedded >> ThreadStackSize >> >> But I need to see where you stick the WSGI directives in relation to >> other stuff you have in the VirtualHost to check whether the location is >> correct and thus whether applied. >> >> Graham >> >> On 13 Aug 2020, at 3:57 am, Paul Royik <[email protected]> wrote: >> >> So, I watched the video, changed the architecture using Celery. >> Now requests are processed very quickly. >> >> Here is my configuration. >> >> StartServers 1 >> ServerLimit 5 >> MaxRequestWorkers 125 >> MaxConnectionsPerChild 10000 >> >> WSGIDaemonProcess django_app processes=15 threads=5 >> >> >> >> As I understand this allows 75 concurrent requests in mod_wsgi and 125 >> requests in apache. >> >> Is there any reason to make number of concurrent requests in apache higher >> than in mod_wsgi, i.e. 125 vs. 75? Possibly, set it 75 in both cases? I >> don't understand this point. >> >> >> The second question. Things become really faster, but from time to time I >> got 502 error. It appears that apache is killed, because I need to start the >> server manually. >> >> What are the possible reasons for this? RAM, too many requests? Can you give >> me some suggestions? >> >> >> Thank you. >> >> >> On Tuesday, August 11, 2020 at 12:09:19 PM UTC+3, Paul Royik wrote: >>> >>> I'll watch. Thank you! >>> >>> On Tuesday, August 11, 2020 at 11:39:27 AM UTC+3, Graham Dumpleton wrote: >>>> >>>> Did you watch the videos? That is covered in some of them. >>>> >>>> On 11 Aug 2020, at 5:57 pm, Paul Royik <[email protected]> wrote: >>>> >>>> Can you at least tell me how apache settings coordinate with mod_wsgi >>>> settings? >>>> >>>> I set number of threads, processes in Apache and in mod_wsgi. How do >>>> they work together? >>>> >>>> In mod_wsgi processes=5 threads=12. Does it mean 60 concurrent requests? >>>> >>>> On Tuesday, August 11, 2020 at 10:31:56 AM UTC+3, Graham Dumpleton >>>> wrote: >>>>> >>>>> The problem was you had other odd requirements like having non thread >>>>> safe code. >>>>> >>>>> All I can do is suggest you watch these videos. >>>>> >>>>> https://www.youtube.com/watch?v=CPz0s1CQsTE&t=4s >>>>> https://www.youtube.com/watch?v=SGleKfigMsk&t=2s >>>>> https://www.youtube.com/watch?v=k6Erh7oHvns&t=1s >>>>> >>>>> There is no simple answer as one needs to know how your application >>>>> works and have access to performance metrics from an APM service in order >>>>> to tune the server. >>>>> >>>>> Graham >>>>> >>>>> On 11 Aug 2020, at 5:27 pm, Paul Royik <[email protected]> wrote: >>>>> >>>>> One more question. >>>>> >>>>> What is a good configuration if I have 500 concurrent requests and >>>>> there are no long-running tasks? >>>>> >>>>> On Tuesday, August 11, 2020 at 9:52:39 AM UTC+3, Paul Royik wrote: >>>>>> >>>>>> Thank you for your help. >>>>>> >>>>>> On Tuesday, August 11, 2020 at 8:30:27 AM UTC+3, Graham Dumpleton >>>>>> wrote: >>>>>>> >>>>>>> If it is a minor quick running script that does something simple it >>>>>>> should be okay. It is just having long running processes would be more >>>>>>> concerned about. >>>>>>> >>>>>>> On 11 Aug 2020, at 3:18 pm, Paul Royik <[email protected]> wrote: >>>>>>> >>>>>>> You are absolutely correct. Need to change the architecture. >>>>>>> One more question. I also use subprocess.check_output from django. >>>>>>> Is it also bad idea? I'm trying to run a script (non-python) and get it >>>>>>> output. >>>>>>> >>>>>>> On Tuesday, August 11, 2020 at 1:55:51 AM UTC+3, Graham Dumpleton >>>>>>> wrote: >>>>>>>> >>>>>>>> Personally I would be concerned about the architecture you are >>>>>>>> using if you have long running tasks like you describe. It is not >>>>>>>> usually a >>>>>>>> good idea to use 'multiprocessing.Process' to create sub processes >>>>>>>> directly >>>>>>>> from a web server process to perform work. A better architecture would >>>>>>>> be >>>>>>>> to off load the work into a queue using something like Celery and have >>>>>>>> the >>>>>>>> separate job processing system pull the jobs from the queue and >>>>>>>> process >>>>>>>> them. You would also be better off to model the interaction from the >>>>>>>> front >>>>>>>> end as queueing the job and immediately responding with an >>>>>>>> acknowledgement >>>>>>>> to say is queued. The front end can then start polling periodically to >>>>>>>> see >>>>>>>> if the job has finished, and when it has it would get the response >>>>>>>> back. >>>>>>>> The front end can then display the data or save it locally as needed. >>>>>>>> >>>>>>>> This model avoids the problem of requests being parked doing >>>>>>>> nothing for a long time, which with your server configuration is going >>>>>>>> to >>>>>>>> be hugely expensive on memory and not scale very well because of >>>>>>>> limitations of using WSGI process/threading model. You might even >>>>>>>> consider >>>>>>>> not using a WSGI application at all. Instead, use an async web >>>>>>>> application >>>>>>>> paired with Celery for execution of the jobs. Using an async web >>>>>>>> application means you can handle as many parked requests as you want >>>>>>>> and >>>>>>>> they can quite happily sit there waiting for Celery to finish the job >>>>>>>> and >>>>>>>> don't need to use polling. Only thing am not sure about in that is >>>>>>>> what >>>>>>>> async clients there are for Celery. >>>>>>>> >>>>>>>> Graham >>>>>>>> >>>>>>>> On 10 Aug 2020, at 9:09 pm, Paul Royik <[email protected]> wrote: >>>>>>>> >>>>>>>> My django app makes heavy calculations which can be infinite. >>>>>>>> That's why, when user enters the site, i.e. makes a request, heavy >>>>>>>> calculations are wrapped into multiprocessing.Process which runs at >>>>>>>> most 7 >>>>>>>> seconds. >>>>>>>> I can't use threads, because third-party packages are not >>>>>>>> thread-safe. >>>>>>>> >>>>>>>> So, I have around 30 concurrent requests per second. If each >>>>>>>> request can take up to 7 seconds, then it is 30*7=210 concurrent >>>>>>>> requests >>>>>>>> in the worst case. >>>>>>>> Each of these concurrent requests opens multiprocessing.Process, >>>>>>>> which gives (I guess) 210*2=420 (close to 500) concurrent requests in >>>>>>>> the >>>>>>>> worst case. >>>>>>>> That' how I got 500 requests. Possibly, my calculations are >>>>>>>> incorrect. >>>>>>>> >>>>>>>> Average page load time (average response times) is 10 seconds. >>>>>>>> >>>>>>>> I use MPM worker. >>>>>>>> >>>>>>>> I set WSGIProcessGroup >>>>>>>> >>>>>>>> StartServers 100 >>>>>>>> ServerLimit 500 >>>>>>>> >>>>>>>> ThreadsPerChild 1 >>>>>>>> ThreadLimit 1 >>>>>>>> >>>>>>>> MaxRequestWorkers 500 >>>>>>>> MaxConnectionsPerChild 10000 >>>>>>>> >>>>>>>> WSGIApplicationGroup %{GLOBAL} >>>>>>>> WSGIDaemonProcess django_app processes=75 threads=1 python-path='...' >>>>>>>> maximum-requests=10000 request-timeout=20 >>>>>>>> WSGIProcessGroup django_app >>>>>>>> >>>>>>>> WSGIRestrictEmbedded On >>>>>>>> WSGILazyInitialization On >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> What sort of application are you running? >>>>>>>>> >>>>>>>>> What is your average response times? >>>>>>>>> >>>>>>>>> Do you have long running requests, if yes, how long? >>>>>>>>> >>>>>>>>> What Apache MPM are you actually using? >>>>>>>>> >>>>>>>>> My initial impression is that is a quite poor configuration which >>>>>>>>> is only going to chew up huge amounts of memory for no good reason, >>>>>>>>> but I >>>>>>>>> don't know your application requirements. >>>>>>>>> >>>>>>>>> Also, are you even setting WSGIProcessGroup? If it isn't set it >>>>>>>>> makes the whole daemon process configuration moot as it isn't even >>>>>>>>> being >>>>>>>>> used. >>>>>>>>> >>>>>>>>> On 10 Aug 2020, at 7:24 pm, Paul Royik <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> StartServers 50 >>>>>>>>> ServerLimit 200 >>>>>>>>> >>>>>>>>> ThreadsPerChild 1 >>>>>>>>> ThreadLimit 1 >>>>>>>>> >>>>>>>>> MaxRequestWorkers 200 >>>>>>>>> MaxConnectionsPerChild 10000 >>>>>>>>> >>>>>>>>> WSGIApplicationGroup %{GLOBAL} >>>>>>>>> WSGIDaemonProcess process processes=75 threads=1 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Is it enough? Or can it handle only 75 concurrent requests? I don't >>>>>>>>> know how to synchronize apache and mod_wsgi settings. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "modwsgi" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "modwsgi" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "modwsgi" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> >>>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/09d05cdb-3b84-4720-b8b1-4804a57b9d5co%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/09d05cdb-3b84-4720-b8b1-4804a57b9d5co%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > >
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/fe0e6129-dbfb-434a-acfd-13a8ed690437o%40googlegroups.com.
