It looks like it's CGI::QueryExtension#read_body in ActionPack that is the first one to cause problems. When a PUT or POST request arrives, it reads the entire thing into memory. Even if that method is patched, there are tons of other methods that wants to mess around with the request body. According to the documentation, the CGI class is supposed to handle Tempfile objects, but obviously not in this case.
Seems like the way to go is to just use Mongrel and write a HttpHandler as a standalone server without both Rails and Railsdav, at least for a first version. Lots of thanks for pointing me in the right direction. /Daniel On 10/15/07, Tim Kuijsten <[EMAIL PROTECTED]> wrote: > Since a vanilla Mongrel will store uploaded data on disk in a tempfile, maybe > it has something to do with your Railsdav plugin? > > > I just tested an 80 MB upload with my Apache 2.2.6 reverse proxy and Mongrel > 1.0.1 cluster and it does save the object first to /tmp on disk and not in > memory. The memory used by the mongrel process before I started uploading was > 50 MB. While uploading it peaked at 57 MB but was mostly stable around 50-52 > MB. And after the upload was done it was back again at 50 MB. > > > Maybe your $TMPDIR is not writable for the user you are running mongrel under? > > > Just before the upload was finished: > $ ls -hl /tmp/ > total 160288 > -rw------- 1 mongrel wheel 78.2M Oct 15 16:31 mongrel.25.0 > > > > > > I would start looking in the Railsdav plugin you are using, seems like they > changed the default of saving to disk, to memory. > > > > Cheers. > > ----- Original Message ---- > From: Daniel Brahneborg <[EMAIL PROTECTED]> > To: [email protected] > Sent: Monday, October 15, 2007 4:10:06 PM > Subject: Re: [Mongrel] POST with huge HTTP body > > > Thanks for the replies. > > The fact that Mongrel shouldn't be the first one to get the POST/PUT > data > was a good point that I hadn't thought of. However, even when talking > directly to Mongrel, the memory consumption of the application > increases > until it crashes if too much data is sent. In my case, I was sending a > 500MB > file from a Webdav client. At some point all of it was loaded into RAM, > which > doesn't work. > > Are there any more productive alternatives than writing the entire > thing > from scratch in C? Please? :) > > /Daniel > > On 10/15/07, Tim Kuijsten <[EMAIL PROTECTED]> wrote: > > A while ago I wrote a plugin to limit uploads in Mongrel. Mongrel > does save uploads to a tempfile on disk - if the upload is bigger than > about 12 KB - using a Ruby TempFile-object. (which stores in the $TMPDIR, > /tmp on most systems). > > > > > > The request is handed over to Rails after it's fully received by > Mongrel. > > > > > > I'm not sure if this "saving to disk" works the same with chunked > uploads (uploads without a Content-Length header). > > > > > > > > Maybe my plugin can help you: > http://slasaus.netsend.nl/articles/show/7 (warning: it works, but is > not very elegant). > > > > > > Gl, > > > > > > Tim > > > > ----- Original Message ---- > > From: Daniel Brahneborg <[EMAIL PROTECTED]> > > To: [email protected] > > Sent: Monday, October 15, 2007 2:00:20 PM > > Subject: [Mongrel] POST with huge HTTP body > > > > > > Hi, > > > > I found the following page describing how to stream data from the > > server to the client: > > > > http://api.rubyonrails.com/classes/ActionController/Streaming.html > > > > I want to do the opposite, streaming data from the client to the > > server, > > letting the controller saving the data while it's being received. > > In particular, I want to upload large files to the RailsDav > > (http://www.liverail.net/railsdav) plugin, without having to store > > the entire file in memory during the operation. > > > > This seems to be a Mongrel problem, so I started looking at this > > plugin: > > > > http://mongrel.rubyforge.org/docs/upload_progress.html > > > > How do I get Mongrel to store the incoming POST data in a tempfile, > > passing a handler to that file as the raw_post field in the Rails > > request? > > Is it even possible? > > How is the "request" parameter to the process() method related to the > > "request" object in the Rails controllers? > > > > /Daniel > > _______________________________________________ > > Mongrel-users mailing list > > [email protected] > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > > > > > > > > ____________________________________________________________________________________ > > Shape Yahoo! in your own image. Join our Network Research Panel > today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 > > > > > > _______________________________________________ > > Mongrel-users mailing list > > [email protected] > > http://rubyforge.org/mailman/listinfo/mongrel-users > > > _______________________________________________ > Mongrel-users mailing list > [email protected] > http://rubyforge.org/mailman/listinfo/mongrel-users > > > > > > > ____________________________________________________________________________________ > Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, > news, photos & more. > http://mobile.yahoo.com/go?refer=1GNXIC > _______________________________________________ > Mongrel-users mailing list > [email protected] > http://rubyforge.org/mailman/listinfo/mongrel-users > _______________________________________________ Mongrel-users mailing list [email protected] http://rubyforge.org/mailman/listinfo/mongrel-users
