That's all I have in httpd.conf:
ServerRoot "/home/user/webapps/django_app/apache2"
LoadModule alias_module modules/mod_alias.so
LoadModule authz_core_module modules/mod_authz_core.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule wsgi_module modules/mod_wsgi.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule deflate_module modules/mod_deflate.so
RewriteEngine on
RewriteCond %{HTTP_REFERER} motherboard.vice.com [NC]
RewriteRule .* - [F]
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\"" combined
CustomLog /home/user/logs/user/access_django_app.log combined
ErrorLog /home/user/logs/user/error_django_app.log
Listen 18136
KeepAlive Off
SetEnvIf X-Forwarded-SSL on HTTPS=1
MaxRequestWorkers 200
MaxConnectionsPerChild 10000
WSGIDaemonProcess django_app processes=15 threads=5
python-path=/home/user/webapps/django_app:/home/user/webapps/django_app/lib/python3.7
maximum-requests=10000 request-timeout=20
WSGIProcessGroup django_app
WSGIRestrictEmbedded On
WSGILazyInitialization On
WSGIScriptAlias / /path_to_wsgi.py
On Thursday, August 13, 2020 at 12:58:54 AM UTC+3, Graham Dumpleton wrote:
>
> Setting of MaxRequestWorker (MaxClients) has connections to other MPM
> settings, so must see those as well.
>
> Also usually would be no need to set MaxConnectionsPerChild either if
> everything else is setup correctly when using mod_wsgi daemon mode.
>
> Since using mod_wsgi daemon mode there isn't a need to recycle Apache
> child worker processes, nor restrict the Apache child workers so they are
> single threaded
>
> Rather than cut and paste piece meal parts of your configuration, I need
> to see the whole mod_wsgi parts of the configuration, plus all the Apache
> MPM settings.
>
> So need to see all the settings for the following, depending on whether
> using worker or event MPM. The preferred these days is the "event" MPM.
>
> <IfModule mpm_worker_module>
> StartServers 3
> MinSpareThreads 75
> MaxSpareThreads 250
> ThreadsPerChild 25
> MaxRequestWorkers 400
> MaxConnectionsPerChild 0
> </IfModule>
>
> <IfModule mpm_event_module>
> StartServers 3
> MinSpareThreads 75
> MaxSpareThreads 250
> ThreadsPerChild 25
> MaxRequestWorkers 400
> MaxConnectionsPerChild 0
> </IfModule>
>
> Also need to see what you have for:
>
> WSGIDaemonProcess
> WSGIProcessGroup
> WSGIScriptAlias
>
> WSGIRestrictEmbedded
> ThreadStackSize
>
> But I need to see where you stick the WSGI directives in relation to other
> stuff you have in the VirtualHost to check whether the location is correct
> and thus whether applied.
>
> Graham
>
> On 13 Aug 2020, at 3:57 am, Paul Royik <[email protected] <javascript:>>
> wrote:
>
> So, I watched the video, changed the architecture using Celery.
> Now requests are processed very quickly.
>
> Here is my configuration.
>
> StartServers 1
> ServerLimit 5
> MaxRequestWorkers 125
> MaxConnectionsPerChild 10000
>
> WSGIDaemonProcess django_app processes=15 threads=5
>
>
>
> As I understand this allows 75 concurrent requests in mod_wsgi and 125
> requests in apache.
>
> Is there any reason to make number of concurrent requests in apache higher
> than in mod_wsgi, i.e. 125 vs. 75? Possibly, set it 75 in both cases? I don't
> understand this point.
>
>
> The second question. Things become really faster, but from time to time I got
> 502 error. It appears that apache is killed, because I need to start the
> server manually.
>
> What are the possible reasons for this? RAM, too many requests? Can you give
> me some suggestions?
>
>
> Thank you.
>
>
> On Tuesday, August 11, 2020 at 12:09:19 PM UTC+3, Paul Royik wrote:
>>
>> I'll watch. Thank you!
>>
>> On Tuesday, August 11, 2020 at 11:39:27 AM UTC+3, Graham Dumpleton wrote:
>>>
>>> Did you watch the videos? That is covered in some of them.
>>>
>>> On 11 Aug 2020, at 5:57 pm, Paul Royik <[email protected]> wrote:
>>>
>>> Can you at least tell me how apache settings coordinate with mod_wsgi
>>> settings?
>>>
>>> I set number of threads, processes in Apache and in mod_wsgi. How do
>>> they work together?
>>>
>>> In mod_wsgi processes=5 threads=12. Does it mean 60 concurrent requests?
>>>
>>> On Tuesday, August 11, 2020 at 10:31:56 AM UTC+3, Graham Dumpleton wrote:
>>>>
>>>> The problem was you had other odd requirements like having non thread
>>>> safe code.
>>>>
>>>> All I can do is suggest you watch these videos.
>>>>
>>>> https://www.youtube.com/watch?v=CPz0s1CQsTE&t=4s
>>>> https://www.youtube.com/watch?v=SGleKfigMsk&t=2s
>>>> https://www.youtube.com/watch?v=k6Erh7oHvns&t=1s
>>>>
>>>> There is no simple answer as one needs to know how your application
>>>> works and have access to performance metrics from an APM service in order
>>>> to tune the server.
>>>>
>>>> Graham
>>>>
>>>> On 11 Aug 2020, at 5:27 pm, Paul Royik <[email protected]> wrote:
>>>>
>>>> One more question.
>>>>
>>>> What is a good configuration if I have 500 concurrent requests and
>>>> there are no long-running tasks?
>>>>
>>>> On Tuesday, August 11, 2020 at 9:52:39 AM UTC+3, Paul Royik wrote:
>>>>>
>>>>> Thank you for your help.
>>>>>
>>>>> On Tuesday, August 11, 2020 at 8:30:27 AM UTC+3, Graham Dumpleton
>>>>> wrote:
>>>>>>
>>>>>> If it is a minor quick running script that does something simple it
>>>>>> should be okay. It is just having long running processes would be more
>>>>>> concerned about.
>>>>>>
>>>>>> On 11 Aug 2020, at 3:18 pm, Paul Royik <[email protected]> wrote:
>>>>>>
>>>>>> You are absolutely correct. Need to change the architecture.
>>>>>> One more question. I also use subprocess.check_output from django.
>>>>>> Is it also bad idea? I'm trying to run a script (non-python) and get it
>>>>>> output.
>>>>>>
>>>>>> On Tuesday, August 11, 2020 at 1:55:51 AM UTC+3, Graham Dumpleton
>>>>>> wrote:
>>>>>>>
>>>>>>> Personally I would be concerned about the architecture you are using
>>>>>>> if you have long running tasks like you describe. It is not usually a
>>>>>>> good
>>>>>>> idea to use 'multiprocessing.Process' to create sub processes directly
>>>>>>> from
>>>>>>> a web server process to perform work. A better architecture would be to
>>>>>>> off
>>>>>>> load the work into a queue using something like Celery and have the
>>>>>>> separate job processing system pull the jobs from the queue and process
>>>>>>> them. You would also be better off to model the interaction from the
>>>>>>> front
>>>>>>> end as queueing the job and immediately responding with an
>>>>>>> acknowledgement
>>>>>>> to say is queued. The front end can then start polling periodically to
>>>>>>> see
>>>>>>> if the job has finished, and when it has it would get the response
>>>>>>> back.
>>>>>>> The front end can then display the data or save it locally as needed.
>>>>>>>
>>>>>>> This model avoids the problem of requests being parked doing nothing
>>>>>>> for a long time, which with your server configuration is going to be
>>>>>>> hugely
>>>>>>> expensive on memory and not scale very well because of limitations of
>>>>>>> using
>>>>>>> WSGI process/threading model. You might even consider not using a WSGI
>>>>>>> application at all. Instead, use an async web application paired with
>>>>>>> Celery for execution of the jobs. Using an async web application means
>>>>>>> you
>>>>>>> can handle as many parked requests as you want and they can quite
>>>>>>> happily
>>>>>>> sit there waiting for Celery to finish the job and don't need to use
>>>>>>> polling. Only thing am not sure about in that is what async clients
>>>>>>> there
>>>>>>> are for Celery.
>>>>>>>
>>>>>>> Graham
>>>>>>>
>>>>>>> On 10 Aug 2020, at 9:09 pm, Paul Royik <[email protected]> wrote:
>>>>>>>
>>>>>>> My django app makes heavy calculations which can be infinite.
>>>>>>> That's why, when user enters the site, i.e. makes a request, heavy
>>>>>>> calculations are wrapped into multiprocessing.Process which runs at
>>>>>>> most 7
>>>>>>> seconds.
>>>>>>> I can't use threads, because third-party packages are not
>>>>>>> thread-safe.
>>>>>>>
>>>>>>> So, I have around 30 concurrent requests per second. If each request
>>>>>>> can take up to 7 seconds, then it is 30*7=210 concurrent requests in
>>>>>>> the
>>>>>>> worst case.
>>>>>>> Each of these concurrent requests opens multiprocessing.Process,
>>>>>>> which gives (I guess) 210*2=420 (close to 500) concurrent requests in
>>>>>>> the
>>>>>>> worst case.
>>>>>>> That' how I got 500 requests. Possibly, my calculations are
>>>>>>> incorrect.
>>>>>>>
>>>>>>> Average page load time (average response times) is 10 seconds.
>>>>>>>
>>>>>>> I use MPM worker.
>>>>>>>
>>>>>>> I set WSGIProcessGroup
>>>>>>>
>>>>>>> StartServers 100
>>>>>>> ServerLimit 500
>>>>>>>
>>>>>>> ThreadsPerChild 1
>>>>>>> ThreadLimit 1
>>>>>>>
>>>>>>> MaxRequestWorkers 500
>>>>>>> MaxConnectionsPerChild 10000
>>>>>>>
>>>>>>> WSGIApplicationGroup %{GLOBAL}
>>>>>>> WSGIDaemonProcess django_app processes=75 threads=1 python-path='...'
>>>>>>> maximum-requests=10000 request-timeout=20
>>>>>>> WSGIProcessGroup django_app
>>>>>>>
>>>>>>> WSGIRestrictEmbedded On
>>>>>>> WSGILazyInitialization On
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Monday, August 10, 2020 at 1:12:30 PM UTC+3, Graham Dumpleton
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> What sort of application are you running?
>>>>>>>>
>>>>>>>> What is your average response times?
>>>>>>>>
>>>>>>>> Do you have long running requests, if yes, how long?
>>>>>>>>
>>>>>>>> What Apache MPM are you actually using?
>>>>>>>>
>>>>>>>> My initial impression is that is a quite poor configuration which
>>>>>>>> is only going to chew up huge amounts of memory for no good reason,
>>>>>>>> but I
>>>>>>>> don't know your application requirements.
>>>>>>>>
>>>>>>>> Also, are you even setting WSGIProcessGroup? If it isn't set it
>>>>>>>> makes the whole daemon process configuration moot as it isn't even
>>>>>>>> being
>>>>>>>> used.
>>>>>>>>
>>>>>>>> On 10 Aug 2020, at 7:24 pm, Paul Royik <[email protected]> wrote:
>>>>>>>>
>>>>>>>> StartServers 50
>>>>>>>> ServerLimit 200
>>>>>>>>
>>>>>>>> ThreadsPerChild 1
>>>>>>>> ThreadLimit 1
>>>>>>>>
>>>>>>>> MaxRequestWorkers 200
>>>>>>>> MaxConnectionsPerChild 10000
>>>>>>>>
>>>>>>>> WSGIApplicationGroup %{GLOBAL}
>>>>>>>> WSGIDaemonProcess process processes=75 threads=1
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Is it enough? Or can it handle only 75 concurrent requests? I don't
>>>>>>>> know how to synchronize apache and mod_wsgi settings.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "modwsgi" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to [email protected].
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com
>>>>>>>>
>>>>>>>> <https://groups.google.com/d/msgid/modwsgi/bce72a22-5047-4d4d-a7cb-1657672b4d3ao%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "modwsgi" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com
>>>>>>>
>>>>>>> <https://groups.google.com/d/msgid/modwsgi/df05d905-b28c-42ce-bc46-5b754e2ddcbeo%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "modwsgi" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/modwsgi/ce91f94b-9c57-464a-9dd2-79d7ad3184c6o%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "modwsgi" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/modwsgi/50d4fde5-aa0f-483e-8956-8534a485a2d5o%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>>
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/d3807684-557f-4ecb-81c5-754d404346bbo%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>>
>>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com
>
> <https://groups.google.com/d/msgid/modwsgi/60dd7682-f453-43c7-a672-862e8d8ce5aco%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/09d05cdb-3b84-4720-b8b1-4804a57b9d5co%40googlegroups.com.
