Thanks a lot for the detailed information, I now better understand how the parameters are related to one another. I do have monitoring in place, as of course it's a necessity.
Cheers, S Poss Le mar. 2 avr. 2019 à 01:52, Graham Dumpleton <[email protected]> a écrit : > Sorry for the slow reply. Been quite busy trying to finish off some stuff > before a holiday. > > On 28 Mar 2019, at 7:17 pm, Stéphane Poss <[email protected]> wrote: > > Hi, > I am unable to find the correct doc about the problem I have and I hope > you'll forgive my naivety. I'm running a django app using mod_wsgi with > mpm_event. > > The env is: Server Version: Apache/2.4.25 (Debian) OpenSSL/1.0.2r > mod_wsgi/4.6.4 Python/3.5 > > Here are the relevant bits of the Apache VirtualHost conf: > > WSGIScriptAlias / > /opt/virtualenvs/server/lib/python3.5/site-packages/server/wsgi.py > WSGIDaemonProcess server listen-backlog=200 processes=16 > threads=30 display-name=%{GROUP} > python-path=/opt/virtualenvs/server/lib/python3.5/site-packages > restart-interval=3600 graceful-timeout=10 > > > Initial impressions are: > > 1. Threads per daemon process is way too high. Unless you have a highly > I/O wait process, would recommend only 3-5 threads per process and rely on > multiple processes. Having a high number of threads can mean thread > resources go wasted, or requests could pile onto one process and overwhelm > things because of global interpreter lock issues. > > 2. Increasing listener backlog to 200 on daemon process is probably not a > good idea. Even the default of about 100 is way too many usually. Having > backlog there is a bad idea because if your system gets overloaded, you > have a backlog of requests which you still end up handling, yet the user > has probably gone away if delay was significant. Better to reject requests > with an error using queue-timeout rather than put your application in a > permanently overloaded state where never seems to catch up. > > 3. You should not use python-path to refer to a site-packages directory. > Use python-home to refer to the root of the virtual environment instead. > > WSGIApplicationGroup %{GLOBAL} > WSGIProcessGroup server > > I have another Virtual host listening on port 443 with config: > > WSGIScriptAlias / > /opt/virtualenvs/server/lib/python3.5/site-packages/tile_server/wsgi.py > WSGIProcessGroup server > > > the following is the mpm_event config: > > ServerLimit 32 > StartServers 3 > MinSpareThreads 75 > MaxSpareThreads 150 > ThreadLimit 64 > ThreadsPerChild 40 > MaxRequestWorkers 500 > MaxConnectionsPerChild 0 > > > These settings are out of whack. Some general rules about setting these to > avoid strange behaviour. > > 1. MaxRequestWorkers should be an integer multiple of ThreadsPerChild. > > 2. MaxRequestWorkers would normally be ThreadsPerChild * ServerLimit. > > 3. MinSpareThreads should be a multiple of ThreadsPerChild. > > 4. MaxSpareThreads should be a multiple of ThreadsPerChild. > > For (3) and (4), if they aren't a multiple of ThreadsPerChild, you can > invoke strange behaviour that might cause Apache to keep starting/stopping > child processes. > > A better config might be: > > ServerLimit 32 > StartServers 3 > MinSpareThreads 75 > MaxSpareThreads 150 > ThreadLimit 64 > ThreadsPerChild 25 > MaxRequestWorkers 800 > MaxConnectionsPerChild 0 > > I'm having a hard time finding the relationship between the 'processes' > and 'threads' of the WSGIDaemonProcess and the StartServers, > ThreadsPerChild and MaxRequestWorkers of the mpm_event config. I have > checked some of the videos I found on other threads (very interesting!) but > was not able to find the means to understand how to configure the 2 > together. > > > The MPM settings above only related to the Apache child worker processes. > These handle static files requests. For the WSGI application, all they do > is act as a proxy for those requests. > > So MaxRequestWorkers should at least be greater than processes*threads of > daemon process group otherwise the Apache child processes would never have > enough capacity to proxy requests up to the capacity of the daemon process > group. You would add a bit extra capacity in the MPM settings, over what > the daemon process group can handle, so it has capacity to handle static > requests and queued up WSGI application requests. > > What you set the MPM settings and daemon process group settings to depends > on request throughput, and whether WSGI application is CPU and I/O bound. > > You are never going to be able to tune these properly if you don't know > have a way of monitoring throughput, request times, and performance of the > server. > > Bumping up threads in daemon process groups because you think you need to > handle a huge number of concurrent requests, more often than not will just > make things worse and is unnecessary. > > My issue is that I seem not to have a high CPU usage on the host (it's a > VM), using cached data, while not being able to serve more than 60-70 > requests per second. I'm wondering why kind of knob I should turn to > improve the server's usage and thus the request rate. > Another issue I discovered this morning is the following: > > [Thu Mar 28 08:50:53.568321 2019] [wsgi:error] [pid 21253:tid > 140125284001536] (2)No such file or directory: [client > 2a02:121f:21b:0:c6cd:4394:566a:12ea:33664] mod_wsgi (pid=21253): Unable to > connect to WSGI daemon process 'tile-elevation' on > '/var/run/apache2/wsgi.15888.0.1.sock' as user with uid=33. > > Looks like the socket was rotated, but I cannot see why... > > > This is usually because the operating system logging system is force > restarting Apache once a day so it can rotate log files, instead of letting > Apache rotate the log files itself. You can end up seeing this error when > keep alive connections were being used by a client, and the keep alive > connection survived because of graceful restart being used, but daemon > process had been restarted. > > You can avoid this problem by setting the directive: > > WSGISocketRotation Off > > Thanks in advance for the assitance, and thanks for the great tool! > > > Beyond that, it is hard to suggest what you should use without you having > instrumentation for your WSGI application and mod_wsgi so you can monitor > throughput, request times, capacity utilisation etc. > > Do you have any monitoring in place? Have you eliminated that your > bottleneck isn't your Python application code or backend databases etc. > > I would at least suggest using: > > processes=16 threads=5 > > and see what happens. This will eliminate potential issues with the Python > GIL and pilling up of requests in one process. > > If however you are trying to test the setup by using a benchmarking tool > at maximum throughput, you are always going to get silly results. You > should never test a server setup in overloaded state as it tells you > nothing about how to tune it and more often than not just triggers > pathological conditions in the server setup. You want to aim to test with > 40-60 capacity, and set systems up so always running around that much for > typical traffic volumes, scaling horizontally when need more capacity. > > Finally, will mention again the importance of monitoring. If you want to > do this properly, you need it. > > > Cheers! > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
