On 04/02/2015, at 6:22 PM, Paul Royik <[email protected]> wrote:
> Hello. > Is there any way we can proceed? Before we do, I want to make the point that nothing of what we do could help. There is an assumption here on my part that you have yourself at least done something to eliminate the possibility that it is your own application code which is causing the issue. On the basis that you have and you do not believe it is your own code which is causing the problem, I can only steer you away from using Apache 2.2 which has known issues in a couple of areas. These particular issues with Apache 2.2 in particular are as follows: 1. There is no flow control in the Apache child worker processes for response content. What this means is that if a web application is generating a large response a bit at a time, if you have a slow HTTP client which isn't consuming that response quick enough, Apache will start to buffer up that response content in memory at a growing rate. It will not actually block if the amount of buffered response content exceeds some level and so stop itself from going back to the web application to continually get more. This will affect a Python web application under mod_wsgi whether embedded mode or daemon mode is used. The memory growth is always in the Apache child worker process though. So in the case of daemon mode, it isn't actually the daemon process with your web application in it that sees growing memory usage, but the Apache worker child process that is proxying the request to the daemon process. This issue in Apache 2.2 is exacerbated due to how memory pooling works in Apache, where each request thread has its own memory pool, but where there is no default upper bound on the size of that memory pool for each request thread. This means that once a slow HTTP client will cause the persistent memory pool for that one request thread to peak, but memory will not be released back for use by other parts of the application. In a multithread Apache worker, this can in turn happen for every request thread, and so the peak memory usage will occur for each thread in turn if it is common occurrence. So if each request thread peaked at 5MB memory usage and you had 25 threads, that will result in the worker process consuming at least 75MB just because of the lack of flow control. Apache 2.4 fixes this issue and introduces flow control and will block when the accumulated response content exceeds 64Kb, and will not go back to the web application to get more until it has been able to write that buffered content. You therefore don't see this peaking of memory usage per request thread and memory usage stays flat. Although Apache 2.4 fixes the lack of flow control, Apache 2.4 finally now has a threshold on memory pool size and at the end of a request, it will release back memory to the rest of the application for reuse if the threshold is exceeded. To get around the flow control with Apache 2.2, you either have to use Apache 2.4, or use a more recent mod_wsgi version. A recent mod_wsgi helps as it implements its own flow control to get around problems in Apache 2.2. Which ever is used, rather than seeing a worker process balloon out in size to 75MB, then it should stay down in the range of a few MB. 2. The older Apache 1.3 C APIs for reading request content will consume a bit of memory for each read, which isn't released until the end of the request back to the memory pool. This means that if you are dealing with very large request content, and especially if you have a slow HTTP client with data trickling in, you will end up with lots of separate reads of the request content, with each causing a slow growth in memory usage because of memory not being reused immediately. Because of older mod_wsgi versions supporting Apache 1.3, it was using this old API with this issue and so could be affected. With mod_wsgi 4.X support for Apache 1.3 was dropped and allowed a switch to be made to Apache 2.X APIs which didn't have this issue. So this issue could actually technically effect either Apache 2.2 or Apache 2.4 and is only avoided by using a recent mod_wsgi version. If running your Python web application in embedded mode the issue would be exacerbated by the memory pool thresholding issue as explained above. If using daemon mode then the issue is occurring in the daemon mode process. In that case memory pools are disposed of at the end of a request and not reused across requests so issue perhaps not as significant, but still can be an issue. Anyway, the point of explaining these in detail is so that you know and can make a judgment of whether you could even have been affected. If you were not consuming large uploads, nor generating large responses, then any memory issue is unlikely to be due to these. That is where you still need to consider whether it is your own application code. That all said I will end this email, and start dealing with you need to do next in subsequent emails. That you keep cutting out the discussion history from the emails means I have to go search back through old emails to find all the details. Graham -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
