[Catalyst] Handling slow uploads externally
Hello all, I have a CMS-like application using Catalyst for which about 40-50% of requests contain large file uploads. I'm using a pretty standard two-tiered setup to avoid tying up heavy mod_perl procs for slow client downloads. Now I'd like to avoid tying up the heavy procs for slow uploads. Many suggest to run upload handlers as CGI rather than mod_perl because the 1s startup time is negligent compared to the time required to upload. I have tried this and it works, but it still creates a 60+ MB process to sit around for 2 minutes while a dial up user uploads a huge image. I'm thinking of splitting file upload handling out to a simple CGI app (no Catalyst code) that serializes the request, including uploaded files, to a tmp directory. It would then redirect to a Catalyst-handled URL matching the serialized request. Then I can validate the submission in Catalyst and display success or error using my views, etc. Is there something like this in existence already? I searched Google and CPAN but didn't find anything promising. I'd appreciate any tips or suggestions. If I do implement it from scratch, would there be any interest in the CGI and a Catalyst base controller being released on CPAN? Best, Brian Kirkbride ___ List: [EMAIL PROTECTED] Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[EMAIL PROTECTED]/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Handling slow uploads externally
Brian Kirkbride scribbled on 4/12/07 12:16 PM: Is there something like this in existence already? I searched Google and CPAN but didn't find anything promising. I'd appreciate any tips or suggestions. If I do implement it from scratch, would there be any interest in the CGI and a Catalyst base controller being released on CPAN? I'd be interested in collaborating on something if you end up rolling your own. We were looking at this recently: http://trac.lighttpd.net/trac/wiki/Docs%3AModUploadProgress and doing something similar to what you're describing. -- Peter Karman . http://peknet.com/ . [EMAIL PROTECTED] ___ List: [EMAIL PROTECTED] Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[EMAIL PROTECTED]/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Handling slow uploads externally
On 4/12/07, Brian Kirkbride [EMAIL PROTECTED] wrote: Now I'd like to avoid tying up the heavy procs for slow uploads. Many suggest to run upload handlers as CGI rather than mod_perl because the 1s startup time is negligent compared to the time required to upload. I have tried this and it works, but it still creates a 60+ MB process to sit around for 2 minutes while a dial up user uploads a huge image. Yeah, I was reading that and thinking that you'll probably use more memory for CGI, since you still have to run perl but you don't get any copy-on-write benefit. It could be offset by loading as few modules as possible into the CGI, but that seems kind of ugly. A better solution would be a proxy that buffers uploads. I thought mod_proxy would do this for you. Are you certain that it doesn't? If so, take a look at the other proxy options out there. - Perrin ___ List: [EMAIL PROTECTED] Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[EMAIL PROTECTED]/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Handling slow uploads externally
Perrin Harkins wrote: On 4/12/07, Brian Kirkbride [EMAIL PROTECTED] wrote: Now I'd like to avoid tying up the heavy procs for slow uploads. Many suggest to run upload handlers as CGI rather than mod_perl because the 1s startup time is negligent compared to the time required to upload. I have tried this and it works, but it still creates a 60+ MB process to sit around for 2 minutes while a dial up user uploads a huge image. Yeah, I was reading that and thinking that you'll probably use more memory for CGI, since you still have to run perl but you don't get any copy-on-write benefit. It could be offset by loading as few modules as possible into the CGI, but that seems kind of ugly. A better solution would be a proxy that buffers uploads. I thought mod_proxy would do this for you. Are you certain that it doesn't? If so, take a look at the other proxy options out there. - Perrin Perrin, It doesn't look like mod_proxy, which I am currently using, supports this. It looks like Perlbal does however. Any other suggestions? I'd like to avoid Squid as it would be overkill for this application. Thanks! ___ List: [EMAIL PROTECTED] Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[EMAIL PROTECTED]/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Handling slow uploads externally
On 4/12/07, Brian Kirkbride [EMAIL PROTECTED] wrote: It doesn't look like mod_proxy, which I am currently using, supports this. It looks like Perlbal does however. Any other suggestions? Since I always use mod_proxy, I can't vouch for any others. I've seen people mention nginx, pound, and pen. It might also be worth asking about this on the mod_proxy list, or checking the archives. - Perrin ___ List: [EMAIL PROTECTED] Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/[EMAIL PROTECTED]/ Dev site: http://dev.catalyst.perl.org/
