> On 24 Aug 2017, at 8:49 pm, Dave Williams <flashyd...@gmail.com> wrote:
>
> I have an app running under mod_wsgi on a real apache2 instance which takes a
> LONG time to process requests. This is due to huge quantity of database
> operations in a single transaction and is fine for our application
> requirements.
> Due to database locking both the original request (and other requests) have
> to wait. These fail with gateway timeouts. To resolve this I simply increased
> the apache2.conf Timeout setting to the worst case number of seconds.
If using straight Apache/mod_wsgi (not mod_wsgi-express) they shouldn't fail
with gateway timeouts. So can only assume when saying that error message it is
based on what you are know seeing with mod_wsgi-express. The error would be
different if using Apache/mod_wsgi directly and you configured it yourself.
> I now have moved to a docker environment using Graham Dumpleton's excellent
> mod_wsgi-express onbuild container. This uses defaults for apache2 and as a
> result I am back with the same errors.
FWIW, I really don't recommend onbuild Docker images. I am also not putting new
effort into the mod_wsgi-docker image anymore as it isn't built to what I would
regard as best practices. Unfortunately people are dependent on the way it
worked and so can't change it. I have newer images I have been working on which
do things in much better way.
The main problem with onbuild and the mod_wsgi-docker image is that it result
in people running their container as root. This is really not a good idea.
> I have spent a morning wandering around the internet and cannot find out the
> "correct" way to increased the Timeout to the value I require.
> Clearly I could update the prepared httpd config file after it is created but
> this doesnt seem right.
>
> The mod_wsgi-express start-server help talks about various timeouts but I
> never needed to tweak anything on my "real" instance in this area. Are some
> of these relevant to my issue?
> I dont want to randomly start changing things.
>
> Some comments (eg Graham's blogs) indicate httpd settings propagate through
> (via SERVER_ARGS) and the mod_wsgi-start-server script shows it can be set up
> in a .whiskey/server_args file (but no required format given).
> The source code for mod_wsg-express is rather opaque in how it starts the
> httpd instance and under what environment or option switch settings.
You canoeist additional command line arguments for mod_wsgi-express in the
.whiskey/server_args file. These will be added at the end of any builtin
arguments added by the startup script and so can override prior options if need
be. The arguments should be able to be on separate lines if you want.
In your case the main options you are going to want to play with are:
--request-timeout SECONDS
Maximum number of seconds allowed to pass before the
worker process is forcibly shutdown and restarted when
a request does not complete in the expected time. In a
multi threaded worker, the request time is calculated
as an average across all request threads. Defaults to
60 seconds.
--connect-timeout SECONDS
Maximum number of seconds allowed to pass before
giving up on attempting to get a connection to the
worker process from the Apache child process which
accepted the request. This comes into play when the
worker listener backlog limit is exceeded. Defaults to
15 seconds.
--socket-timeout SECONDS
Maximum number of seconds allowed to pass before
timing out on a read or write operation on a socket
and aborting the request. Defaults to 60 seconds.
--queue-timeout SECONDS
Maximum number of seconds allowed for a request to be
accepted by a worker process to be handled, taken from
the time when the Apache child process originally
accepted the request. Defaults to 30 seconds.
--response-socket-timeout SECONDS
Maximum number of seconds allowed to pass before
timing out on a write operation back to the HTTP
client when the response buffer has filled and data is
being forcibly flushed. Defaults to 0 seconds
indicating that it will default to the value of the
'socket-timeout' option.
You may also need to use:
--processes NUMBER The number of worker processes (instances of the WSGI
application) to be started up and which will handle
requests concurrently. Defaults to a single process.
--threads NUMBER The number of threads in the request thread pool of
each process for handling requests. Defaults to 5 in
each process.
--max-clients NUMBER The maximum number of simultaneous client connections
that will be accepted. This will default to being 1.5
times the total number of threads in the request
thread pools across all process handling requests.
Questions I would need answers to make recommends are:
* How long does a request actually take up to?
* How many processes are you running? It will be 1 if you didn't change it.
* How many threads are you running in that process for handling requests? It
will be 5 if you didn't change it.
* How many concurrent requests do you expect?
For an I/O bound application, first thing you might do is increase the number
of threads. So add to 'server_args' file:
--threads 20
This gives you more capacity to handle concurrent requests.
If you still have a various CPU bound activities, you should balance things out
by increasing the number of processes at same time.
--threads 10
--processes 2
Either way, it allows you to have more requests in the Python application
waiting directly on the database operation to finish.
If the number of concurrent requests exceeds capacity (threads*processes), then
requests will queue up.
If those requests are queued for more than the queue timeout, they will be
failed before even being handled and you will see the gateway timeout for those.
This first preference is to increase the capacity because timing out requests
in the queue is important as far as recovering a server which temporarily gets
stuck and too many requests backlog. If your clients are users, they have
probably given up after a while and no point handling a request which has been
sitting there a long time when you are already trying to catch up because you
reached capacity.
If you still want to allow requests to be queue for longer, then use:
--queue-timeout 30
Make that too large such that requests backlog into the Apache child processes,
then you may want to increase maximum number of clients if happy with that.
--max-clients 40
This defaults to 1.5*threads*processes. When you set it you need to work out a
value yourself. It should be greater than threads*processes.
As to the Timeout directive, that is a base timeout applying to any single read
or write operation on a socket. It can be modified using --socket-timeout, or
for specific sub cases the --response-socket-timeout option.
You want to be a little more careful playing with that one. It really depends
on how long your request is going to take and needs to be considered in
relation to --request-timeout as well.
The request timeout is a notional cap on how long a request is allowed to run
before it is failed. How it works is a bit tricky though. In a threaded
process, the request timeout is actually an average across all request threads
for active requests. So although it says 60 seconds, a request can actually run
longer if there was no other requests running and so was still free capacity.
It would only kick in at 60 seconds if you had concurrent requests all start at
the same time and they all hit 60 seconds running time. This is why really need
to understand how long your requests actually run for before can say what to do
about it.
Graham
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to modwsgi+unsubscr...@googlegroups.com.
To post to this group, send email to modwsgi@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
