On 19/10/2014, at 7:59 AM, Laurence Rowe <[email protected]> wrote:
> My application code is a little leaky. Someday I hope to fix this, but in the > meantime I have resorted to checking the process memory usage after a request > has completed using the ExecuteOnCompletion2 wsgi filter from > https://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode > > def rss_checker(rss_limit=None): > log = logging.getLogger(__name__) > process = psutil.Process() > > def callback(environ): > rss = process.memory_info().rss > if rss_limit and rss > rss_limit: > msg = "Restarting process. Memory usage exceeds limit of %d: %d" > log.error(msg, rss_limit, rss) > process.kill() > > return callback > > This seems to work well enough, mod_wsgi restarts the process for me and > service is uninterrupted. It has an obvious limitation though in that it can > only work when WSGIDaemonProcess threads=1. Could there be some mechanism to > communicate to mod_wsgi that it should restart a daemon process gracefully, > once all existing in-flight requests are served? I guess my ideal solution > might involve a watchdog thread in my application that periodically checked > resource usage and signalled mod_wsgi when it desired a graceful restart. > > I measure resource usage rather than setting `maximum-requests` because > different usage patterns have very different memory usage profiles. Have a look at the example code for monitoring source code changes and triggering a restart in: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode#Monitoring_For_Code_Changes Instead of checking for source code changes, you would check memory usage against some threshold. That uses SIGINT to signal the daemon process should restart. When a SIGINT is used, then if there are no currently executing requests, the restart will happen immediately. If there are executing requests, they will be given a small grace period dependent on the shutdown timeout. This defaults to 5 seconds. The WSGIDaemonProcess option for this is shutdown-timeout if for some reason you needed to override it, but in general setting it to be longer would be ill advised. So, if all currently executing requests complete within 5 seconds of the signal being received, the process will then be restarted. If there were still active requests at the end of 5 seconds, the process will be forcibly shutdown. During the period of the shutdown timeout, no new requests will be accepted by that daemon process. No new requests are accepted so as to avoid them delaying the shutdown if the currently executing requests are able to complete. The shutdown timeout is only 5 seconds because if you were to wait indefinitely and a request never completes, that no new requests are being accepted means the whole process would get stuck at that point. The shutdown timeout is therefore a failsafe. If you are stuck with an ancient mod_wsgi version 3.X because of only wanting to use a Linux distributions packaged versions, then that is all you have. If however you are using newer mod_wsgi version 4.X, a slightly more graceful restart option is available. What you can do in mod_wsgi version 4.X is set the graceful-timeout option on WSGIDaemonProcess. You might for example set it to 30 seconds. Now, rather than send a SIGINT to a daemon process, send a SIGUSR1 instead. What will happen this time is that if there are no currently active requests, then the process will restart immediately. If there are currently active requests then they will again be given time to complete within the timeout period and when they complete, the process will be restarted. Things will be slightly different this time though with the graceful-timeout option set, with there being two timeout periods in play. For the first 30 seconds as specified by what you set graceful-timeout to, then new requests will still be accepted. As before though, if the process becomes idle with no currently active requests, then the process will restart. So the graceful timeout period allows you to specify a much longer period of time to allow any active requests to complete, but at the same time allowing the process to still allow new requests. Unless the process is accepting a lot of requests, you will usually see holes where there are no requests being handled so it can restart. If however you have a high throughput, stuck requests, or very long running requests, then that isn't going to happen. Instead, at the end of the graceful-timeout period, then it will go into the shutdown sequence as if SIGINT had been received. That is, no new requests accepted and a shutdown timeout of 5 seconds applied. If the requests still didn't complete within that further 5 seconds, then the process will be forcibly restarted. The graceful-timeout option in mod_wsgi 4.X is just one a of a number of new mechanisms for controlling process restart, either explicitly in this case, or due to process inactivity, queue backlogging, or because of requests getting blocked resulting in the capacity of the process starting to decline. BTW, be a little careful in how strict you apply the memory limit. These days Python can make use of high memory allocations for large contiguous blocks of memory. If when such large blocks of memory are released, there had been no further new allocations above it from the operating system, that high memory can actually be given back to the operating system. Thus you technically could see memory usage jump up, but then go back down. So see how it plays out in your specific web application, but if what you are doing might see such high memory allocations and releases, perhaps only restart if the process exceeds some lower limit for 30 seconds, but where if it goes over some higher limit, only then does it restart straight away. The delay will accommodate for such fluctuations in memory usage, as such transient high memory allocations are not usually going to last too long if they do occur. Graham -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
