Incremental memory use when performing streaming over long period.

                 Key: MODPYTHON-241
             Project: mod_python
          Issue Type: Bug
          Components: core
    Affects Versions: 3.3.1, 3.2.10
            Reporter: Graham Dumpleton
            Priority: Minor

As first raised in:

when streaming data from mod_python handler over a long period of time, memory 
use will keep increasing.

Note that this isn't actually a memory leak though, as the memory gets 
reclaimed at the end of the request.

The underlying problem arises from mod_python using the ap_rwrite() function to 
write output data. When this function is used it will create a bucket chain for 
each block of data written, with the bucket chain coming from the request 
memory pool. There is no attempt to reuse the bucket chain for successive 
writes, so memory will increase on every call.

The normal solution to this problem is to retain the bucket chain and use it 
from one call to the next. In mod_python though this isn't simple to achieve 
because calls to response data may or may not be flushed at the discretion of 
the user code. The retaining of the bucket chain and using it from one call to 
the next generally can only be done when data is always being flushed. A more 
complicated system would be needed which can cope with a need to buffer data at 
times and yet flush at other times.

One approach may be to not actually use Apache itself to buffer data between 
flushes, but to use Python data structures instead to accumulate references to 
Python string objects in a Python list so objects are still valid. When flushed 
only then construct a chain of buckets which use direct reference to raw data 
of Python string objects.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to