Is there a reason you discarded the default Apache configuration? It is important you do not do that as it contains defaults that keep your system more secure. By deleting all the default config, you have lessoned the security of your system.
Also, when you delete the configuration, you don't end up with the same defaults for many values. This is because the compiled in defaults are often not the same as is set in the files. It is therefore going to be far from clear what configuration your system is even running with at this point. Graham > On 14 Aug 2020, at 12:09 am, Paul Royik <[email protected]> wrote: > > That's all I have in httpd.conf: > > ServerRoot "/home/user/webapps/django_app/apache2" > > LoadModule alias_module modules/mod_alias.so > LoadModule authz_core_module modules/mod_authz_core.so > LoadModule dir_module modules/mod_dir.so > LoadModule env_module modules/mod_env.so > LoadModule log_config_module modules/mod_log_config.so > LoadModule mime_module modules/mod_mime.so > LoadModule rewrite_module modules/mod_rewrite.so > LoadModule setenvif_module modules/mod_setenvif.so > LoadModule wsgi_module modules/mod_wsgi.so > LoadModule unixd_module modules/mod_unixd.so > LoadModule deflate_module modules/mod_deflate.so > > RewriteEngine on > RewriteCond %{HTTP_REFERER} motherboard.vice.com > <http://motherboard.vice.com/> [NC] > RewriteRule .* - [F] > > LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" > \"%{User-Agent}i\"" combined > CustomLog /home/user/logs/user/access_django_app.log combined > ErrorLog /home/user/logs/user/error_django_app.log > > Listen 18136 > KeepAlive Off > SetEnvIf X-Forwarded-SSL on HTTPS=1 > > MaxRequestWorkers 200 > MaxConnectionsPerChild 10000 > > WSGIDaemonProcess django_app processes=15 threads=5 > python-path=/home/user/webapps/django_app:/home/user/webapps/django_app/lib/python3.7 > maximum-requests=10000 request-timeout=20 > WSGIProcessGroup django_app > WSGIRestrictEmbedded On > WSGILazyInitialization On > WSGIScriptAlias / /path_to_wsgi.py > > > > On Thursday, August 13, 2020 at 12:58:54 AM UTC+3, Graham Dumpleton wrote: > Setting of MaxRequestWorker (MaxClients) has connections to other MPM > settings, so must see those as well. > > Also usually would be no need to set MaxConnectionsPerChild either if > everything else is setup correctly when using mod_wsgi daemon mode. > > Since using mod_wsgi daemon mode there isn't a need to recycle Apache child > worker processes, nor restrict the Apache child workers so they are single > threaded > > Rather than cut and paste piece meal parts of your configuration, I need to > see the whole mod_wsgi parts of the configuration, plus all the Apache MPM > settings. > > So need to see all the settings for the following, depending on whether using > worker or event MPM. The preferred these days is the "event" MPM. > > <IfModule mpm_worker_module> > StartServers 3 > MinSpareThreads 75 > MaxSpareThreads 250 > ThreadsPerChild 25 > MaxRequestWorkers 400 > MaxConnectionsPerChild 0 > </IfModule> > > <IfModule mpm_event_module> > StartServers 3 > MinSpareThreads 75 > MaxSpareThreads 250 > ThreadsPerChild 25 > MaxRequestWorkers 400 > MaxConnectionsPerChild 0 > </IfModule> > > Also need to see what you have for: > > WSGIDaemonProcess > WSGIProcessGroup > WSGIScriptAlias > > WSGIRestrictEmbedded > ThreadStackSize > > But I need to see where you stick the WSGI directives in relation to other > stuff you have in the VirtualHost to check whether the location is correct > and thus whether applied. > > Graham > >> On 13 Aug 2020, at 3:57 am, Paul Royik <distan...@ <>gmail.com >> <http://gmail.com/>> wrote: >> >> So, I watched the video, changed the architecture using Celery. >> Now requests are processed very quickly. >> >> Here is my configuration. >> StartServers 1 >> ServerLimit 5 >> MaxRequestWorkers 125 >> MaxConnectionsPerChild 10000 >> >> WSGIDaemonProcess django_app processes=15 threads=5 >> >> >> As I understand this allows 75 concurrent requests in mod_wsgi and 125 >> requests in apache. >> Is there any reason to make number of concurrent requests in apache higher >> than in mod_wsgi, i.e. 125 vs. 75? Possibly, set it 75 in both cases? I >> don't understand this point. >> >> The second question. Things become really faster, but from time to time I >> got 502 error. It appears that apache is killed, because I need to start the >> server manually. >> What are the possible reasons for this? RAM, too many requests? Can you give >> me some suggestions? >> >> Thank you. >> >> On Tuesday, August 11, 2020 at 12:09:19 PM UTC+3, Paul Royik wrote: >> I'll watch. Thank you! >> >> On Tuesday, August 11, 2020 at 11:39:27 AM UTC+3, Graham Dumpleton wrote: >> Did you watch the videos? That is covered in some of them. >> >>> On 11 Aug 2020, at 5:57 pm, Paul Royik <distan...@ <>gmail.com >>> <http://gmail.com/>> wrote: >>> >>> Can you at least tell me how apache settings coordinate with mod_wsgi >>> settings? >>> >>> I set number of threads, processes in Apache and in mod_wsgi. How do they >>> work together? >>> >>> In mod_wsgi processes=5 threads=12. Does it mean 60 concurrent requests? >>> >>> On Tuesday, August 11, 2020 at 10:31:56 AM UTC+3, Graham Dumpleton wrote: >>> The problem was you had other odd requirements like having non thread safe >>> code. >>> >>> All I can do is suggest you watch these videos. >>> >>> https://www.youtube.com/watch?v=CPz0s1CQsTE&t=4s >>> <https://www.youtube.com/watch?v=CPz0s1CQsTE&t=4s> >>> https://www.youtube.com/watch?v=SGleKfigMsk&t=2s >>> <https://www.youtube.com/watch?v=SGleKfigMsk&t=2s> >>> https://www.youtube.com/watch?v=k6Erh7oHvns&t=1s >>> <https://www.youtube.com/watch?v=k6Erh7oHvns&t=1s> >>> >>> There is no simple answer as one needs to know how your application works >>> and have access to performance metrics from an APM service in order to tune >>> the server. >>> >>> Graham >>> >>>> On 11 Aug 2020, at 5:27 pm, Paul Royik <distan...@ <>gmail.com >>>> <http://gmail.com/>> wrote: >>>> >>>> One more question. >>>> >>>> What is a good configuration if I have 500 concurrent requests and there >>>> are no long-running tasks? >>>> >>>> On Tuesday, August 11, 2020 at 9:52:39 AM UTC+3, Paul Royik wrote: >>>> Thank you for your help. >>>> >>>> On Tuesday, August 11, 2020 at 8:30:27 AM UTC+3, Graham Dumpleton wrote: >>>> If it is a minor quick running script that does something simple it should >>>> be okay. It is just having long running processes would be more concerned >>>> about. >>>> >>>>> On 11 Aug 2020, at 3:18 pm, Paul Royik <distan...@ <>gmail.com >>>>> <http://gmail.com/>> wrote: >>>>> >>>>> You are absolutely correct. Need to change the architecture. >>>>> One more question. I also use subprocess.check_output from django. Is it >>>>> also bad idea? I'm trying to run a script (non-python) and get it output. >>>>> >>>>> On Tuesday, August 11, 2020 at 1:55:51 AM UTC+3, Graham Dumpleton wrote: >>>>> Personally I would be concerned about the architecture you are using if >>>>> you have long running tasks like you describe. It is not usually a good >>>>> idea to use 'multiprocessing.Process' to create sub processes directly >>>>> from a web server process to perform work. A better architecture would be >>>>> to off load the work into a queue using something like Celery and have >>>>> the separate job processing system pull the jobs from the queue and >>>>> process them. You would also be better off to model the interaction from >>>>> the front end as queueing the job and immediately responding with an >>>>> acknowledgement to say is queued. The front end can then start polling >>>>> periodically to see if the job has finished, and when it has it would get >>>>> the response back. The front end can then display the data or save it >>>>> locally as needed. >>>>> >>>>> This model avoids the problem of requests being parked doing nothing for >>>>> a long time, which with your server configuration is going to be hugely >>>>> expensive on memory and not scale very well because of limitations of >>>>> using WSGI process/threading model. You might even consider not using a >>>>> WSGI application at all. Instead, use an async web application paired >>>>> with Celery for execution of the jobs. Using an async web application >>>>> means you can handle as many parked requests as you want and they can >>>>> quite happily sit there waiting for Celery to finish the job and don't >>>>> need to use polling. Only thing am not sure about in that is what async >>>>> clients there are for Celery. >>>>> >>>>> Graham >>>>> >>>>>> On 10 Aug 2020, at 9:09 pm, Paul Royik <distan...@ <>gmail.com >>>>>> <http://gmail.com/>> wrote: >>>>>> >>>>>> My django app makes heavy calculations which can be infinite. >>>>>> That's why, when user enters the site, i.e. makes a request, heavy >>>>>> calculations are wrapped into multiprocessing.Process which runs at most >>>>>> 7 seconds. >>>>>> I can't use threads, because third-party packages are not thread-safe. >>>>>> >>>>>> So, I have around 30 concurrent requests per second. If each request can >>>>>> take up to 7 seconds, then it is 30*7=210 concurrent requests in the >>>>>> worst case. >>>>>> Each of these concurrent requests opens multiprocessing.Process, which >>>>>> gives (I guess) 210*2=420 (close to 500) concurrent requests in the >>>>>> worst case. >>>>>> That' how I got 500 requests. Possibly, my calculations are incorrect. >>>>>> >>>>>> Average page load time (average response times) is 10 seconds. >>>>>> >>>>>> I use MPM worker. >>>>>> >>>>>> I set WSGIProcessGroup >>>>>> >>>>>> StartServers 100 >>>>>> ServerLimit 500 >>>>>> >>>>>> ThreadsPerChild 1 >>>>>> ThreadLimit 1 >>>>>> >>>>>> MaxRequestWorkers 500 >>>>>> MaxConnectionsPerChild 10000 >>>>>> >>>>>> WSGIApplicationGroup %{GLOBAL} >>>>>> WSGIDaemonProcess django_app processes=75 threads=1 python-path='...' >>>>>> maximum-requests=10000 request-timeout=20 >>>>>> WSGIProcessGroup django_app >>>>>> WSGIRestrictEmbedded On >>>>>> WSGILazyInitialization On >>>>>> >>>>>> >>>>>> On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton wrote: >>>>>> What sort of application are you running? >>>>>> >>>>>> What is your average response times? >>>>>> >>>>>> Do you have long running requests, if yes, how long? >>>>>> >>>>>> What Apache MPM are you actually using? >>>>>> >>>>>> My initial impression is that is a quite poor configuration which is >>>>>> only going to chew up huge amounts of memory for no good reason, but I >>>>>> don't know your application requirements. >>>>>> >>>>>> Also, are you even setting WSGIProcessGroup? If it isn't set it makes >>>>>> the whole daemon process configuration moot as it isn't even being used. >>>>>> >>>>>>> On 10 Aug 2020, at 7:24 pm, Paul Royik <distan...@ <>gmail.com >>>>>>> <http://gmail.com/>> wrote: >>>>>>> >>>>>>> StartServers 50 >>>>>>> ServerLimit 200 >>>>>>> >>>>>>> ThreadsPerChild 1 >>>>>>> ThreadLimit 1 >>>>>>> >>>>>>> MaxRequestWorkers 200 >>>>>>> MaxConnectionsPerChild 10000 >>>>>>> >>>>>>> WSGIApplicationGroup %{GLOBAL} >>>>>>> WSGIDaemonProcess process processes=75 threads=1 >>>>>>> >>>>>>> >>>>>>> Is it enough? Or can it handle only 75 concurrent requests? I don't >>>>>>> know how to synchronize apache and mod_wsgi settings. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "modwsgi" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "modwsgi" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "modwsgi" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mod...@ <>googlegroups.com <http://googlegroups.com/>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/09d05cdb-3b84-4720-b8b1-4804a57b9d5co%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/09d05cdb-3b84-4720-b8b1-4804a57b9d5co%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/769C94B8-EC5C-479F-913C-EB30D8AD4EA7%40gmail.com.
