> On 19 Nov 2019, at 2:53 am, Phivos Stylianides <[email protected]> wrote: > > I have a backend service which exposes a single endpoint to perform some > database updates. Each request is expected to take between 5-25 seconds on > average. Apache/mod_wsgi and the flask app sit on 2 Elastic Beanstalk > instances with 2 CPUs and 8GB ram each. I want to limit the traffic send to > the app to 10-20 requests at any given time per instance.
Currently you are using: processes=10 threads=5 which means the application can handle up to 50 requests at any one time. Up to 5 requests per process, with 10 processes. If you want to cap it at a lower maximum number, you need to change those. > Any additional requests should queue in the backlog and if not picked up > within the configured timeout they should be dropped until repeated later. > > For some reason the number of requests per process goes really high, and so > the average latency starts to go up and failure rates increase too. I setup > recording of `mod_wsgi.process_metrics()["request_count"]` stats and I > verified it goes from 0 to >180, and > `mod_wsgi.process_metrics()["threads"]["request_count"]` ranging from 24-65 > within less than 1 hour. The request_count values from memory are an incrementing counter of number of requests over time, so it will keep increasing. It is not a count of the number of current active requests. You would need to track len(mod_wsgi.active_requests) on a per process basis to know how many are active at the time of the call. > I tried reducing Apache's `MaxRequestWorkers` and `ListenBackLog`. Also > WSGIDaemonProcess' `queue-timeout` and `listen-backlog` but no luck. It's > important to not accept a request when max capacity is reached and then > timeout halfway while processing because all threads were too busy. This is a > worker service so it's fine to drop the request as it will be retried later. It is unclear what you want to happen. FWIW, you may want to read: * http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html <http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html> If you want really fine grained control over what happens when back logging occurs, you probably can't use a WSGI server, and should write an async HTTP application to handle requests. That way you can hold requests in your own queue if you want to try and limit number of concurrent database updates. > I'm using Apache 2.4.39 and mod_wsgi 4.6.5. Apache is configured as follows: > > <IfModule reqtimeout_module> > RequestReadTimeout header=15,MinRate=500 body=150,MinRate=500 > </IfModule> > > > TimeOut 150 > > > KeepAlive Off > KeepAliveTimeOut 0 > > > <IfModule mpm_worker_module> > StartServers 1 > ServerLimit 2 > MinSpareThreads 5 > MaxSpareThreads 10 > ThreadLimit 5 > ThreadsPerChild 5 > ListenBackLog 10 > MaxRequestWorkers 10 > MaxConnectionsPerChild 5000 > </IfModule> > > mod_wsgi config: > > WSGIDaemonProcess wsgi \ > processes=10 \ > threads=5 \ > display-name=%{GROUP} \ > python-home=/opt/python/run/venv/ \ > python-path=/opt/python/current/app user=wsgi group=wsgi \ > home=/opt/python/current/app \ > lang='en_US.UTF-8' \ > locale='en_US.UTF-8' \ > connect-timeout=15 \ > socket-timeout=25 \ > request-timeout=25 \ > deadlock-timeout=120 \ > graceful-timeout=120 \ > restart-interval=0 \ > maximum-requests=200 \ > queue-timeout=15 \ > listen-backlog=10 > > > WSGIProcessGroup wsgi > WSGIApplicationGroup %{GLOBAL} > > Any ideas what might do the trick? > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/f386576b-2e19-4115-9ecb-f954a389be02%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/f386576b-2e19-4115-9ecb-f954a389be02%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/4D2DF949-A307-485A-9EAF-391E6F3CD8A0%40gmail.com.
