On Thu, Nov 19, 2009 at 2:25 AM, John Buckman <j...@magnatune.com> wrote: > I'm writing an app that accepts large file uploads, and trying to lower the > memory requirements. > > When I upload large files, at the point where the files are completely > uploaded, Naviserver consumes an equivalent amount of memory as the file > uploaded size, as it hands control over to my form handler. Memory is > released once my form handler returns. Virtually no memory is used (thanks, > spooler) as the large file is being uploaded, only once the upload completes. > > I'm not sure why Naviserver needs to hold the file in memory. > > So, questions: > > a) is this by design? > b) has anyone on this list worked on this section of code? > c) any fixes, or should I plunge into the nsd/form.c code (is that right?) to > try to fix this?
We originally talked about this way back in 2005: http://sourceforge.net/mailarchive/message.php?msg_id=37583D44-1F17-401D-8BCA-26F816831B92%40archiware.com Spooling got implemented later. What didn't didn't get implemented, and what you're tripping over, is the on-the-fly parsing of file-upload mime headers so that *only* the contents of the actual file becomes a file spooled on disk, and not the file topped and tailed with mime headers. As you've discovered, the way the form parsing is implemented it just reads the whole thing back into memory anyway, negating the benefit of spooling to disk. Ideally, the spooler thread would parse the form as it is read in chunks from the socket, placing variables in the form structure, and spooling file contents to disk. If more than one file is in the form then multiple spool files would be used. Watch out for encoding. IIRC the way it currently works forms can be reparsed if the encoding changes. Perhaps everything but the file parts of a multipart form could be saved in a buffer..? ie. you strip out the large files from the input stream and leave the rest for normal form processing. Also watch out for which temp directory files are being spooled to. If it is on a separate partition then the final rename() will actually be a file copy. Large files should probably be sent to a configured spool directory. ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ naviserver-devel mailing list naviserver-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/naviserver-devel
