[EMAIL PROTECTED] wrote .. > > > Graham, > Thank you for your reply. > > Previously you said you were running > > out of resources which suggested runaway memory use. Ie., process size > > just kept growing and growing until machine memory exhausted. > When I used previously my program with django handler ( handler supplied > with Django), > it was the case. > Now when I use my own handler the memory increases, as I described before, > but possibly it > is as wrote and it could be acceptable. > But is there any advantage in using input filter if memory still increases? > Or would be the situation the same ( in memory using) if I use ONLY my > own handler?
In certain cases using an input filter to intercept large file uploads can be beneficial, however, the FieldStorage class in newer versions of mod_python has the ability to provide callback hooks to control where uploaded files are placed when using FieldStorage. If you are writing your own handler, FieldStorage may thus be sufficient. Check out the mod_python documentation for FieldStorage. You might also check out Tramline. It is a system for dealing with large file uploads when Apache is just a front end for some back end application to which it proxies requests. It probably will not do exactly what you want, but you might learn from it. Tramline can be found at: http://www.infrae.com/products/tramline BTW, please keep discussion on the mailing list. Graham > Thank you for reply > Lad > > > > > > On 17/01/2007, at 8:54 PM, [EMAIL PROTECTED] wrote: > > > > > > > > Thank you for your reply. > > > Now I use > > > Apache 2.2.4 > > > Mod_python 3.3.0.b > > > Python 2.3.5 > > > The results are more or less the same when I used > > > Apache 2.0 > > > and mod_python 3.1.3-1 > > > Python 2.3.4 > > > > > > Memory increases when I read (filter.read()) or write( filter.write > > > (s). > > > In otherwords, memory is not released after I wrote data to a file > > > ( before next reading). > > > > > > Is my presumption correct that the memory should be released > > > during each > > > reading/writing in the filter? > > > Or do I make a mistake somewhere? > > > > You are giving conflicting information. Previously you said you were > > running > > out of resources which suggested runaway memory use. Ie., process size > > just kept growing and growing until machine memory exhausted. Is that > > what > > is happening or were you more echoing your concerns rather than what > was > > happening. > > > > What you will probably see after having restarted Apache and then > > doing an > > initial upload is that memory use by Apache will grow as it reads in > > more and > > more of the file. Once the complete request has been serviced, Apache > > will > > continue to use at least as much memory as it allocated by that > > point. You > > will not see the Apache process size shrink as once the Apache process > > has allocated memory, it notionally does not give it back to the > > operating system > > until the process has quit. This doesn't mean it is lost though as it > > is put on > > a free list within the Apache process itself and when a subsequent > > upload of > > same file for example occurs, it will just use that free memory. > > Thus, process > > size will grow on the first request but if you keep doing the same > > request, the > > process size should stay reasonably stable. > > > > Note that how much the Apache process grows to initially can depend a > > bit > > on how a handler deals with a file upload. If the response handler > > does a > > req.read() and then processes it all in one go, it will have to hold > > the whole > > file in memory and so if it is a big file it will use a lot. If > > instead the response > > handler read in the file in small blocks at a time using req.read > > (blocksize) > > and process each block and then discarded it, the memory use by Apache > > will be less as it doesn't have to hold the whole file in memory but > > more or > > less just what it is processing. > > > > It is hard to tell if you do have a problem with a memory leak or > > whether you > > just understand how memory allocation works for processes. > > > > What are you using to monitor memory use? Does memory use keep growing > > continually after every request or does it plateau after the initial > > increases > > with the firs requests? > > > > Graham > > > > > Thank you for help > > > > > > Regards, > > > Lad. > > > > > > > > > > On 16/01/2007, at 6:54 PM, [EMAIL PROTECTED] wrote: > > > > > > > > > I have this, very simple input filter > > > > > > > > > > def inputfilter(filter): > > > > > if filter.req.method != 'POST': > > > > > filter.pass_on() > > > > > return > > > > > filter.req.log_error('first read') > > > > > s = filter.read() > > > > > while s: > > > > > filter.req.log_error('writing (%s)' % len(s)) > > > > > filter.write(s) > > > > > #filter.flush() > > > > > s = filter.read() > > > > > if s is None: > > > > > filter.req.log_error('closing') > > > > > filter.close() > > > > > > > > > > I would like to use an input filter for uploading large files. > > > > > I would expect that after each writing/reading, filter releases > > > > > memory.But instead memory > > > > > increases during reading/writing and no memory is released > > > after each > > > > > reading/writing. Is it a common feature that the filter during > > > > > reading/writing does NOT release memory and memory increases > > > until the > > > > > whole file is uploaded? Or is my presumption correct that the > > > memory > > > > > should be released during each reading/writing in the filter? > > > Thank you > > > > > for help and reply. > > > > > > > > Since I don't recollect you saying what version of mod_python you > > > are > > > > using, I would suggest you upgrade to mod_python 3.3.0b if you > > > aren't > > > > using it already. Once you have confirmed that you still have > > > the problem > > > > with that version let us know. Also state which versions of > > > Apache and > > > > Python you are using at the same time. > > > > > > > > For 3.3.0b see: > > > > > > > > http://httpd.apache.org/modules/python-download.cgi > > > > http://nicolas.lehuen.com/download/mod_python/ > > > > > > > > The latter has Windows binaries for various Apache/Python combos. > > > > > > > > Graham > > > > > > > ------- End of forwarded message -------
