Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-10 Thread Paul Winkler
On Fri, Apr 09, 2004 at 08:11:20PM -0400, Chris McDonough wrote: On Fri, 2004-04-09 at 18:02, Paul Winkler wrote: That's easy. LocalFS, CMFCore.FSFile, and ExtFile 1.1.3 all read the entire file into memory before sending anything back to the client. That's why ExtFile 1.4 is so much better

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-09 Thread Paul Winkler
On Fri, Apr 09, 2004 at 12:25:50AM -0400, Chris McDonough wrote: and 1216K (I cannot explain the large difference in results at 38322k across products; it appears to be product-specific). That's easy. LocalFS, CMFCore.FSFile, and ExtFile 1.1.3 all read the entire file into memory before sending

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-09 Thread Chris McDonough
On Fri, 2004-04-09 at 18:02, Paul Winkler wrote: That's easy. LocalFS, CMFCore.FSFile, and ExtFile 1.1.3 all read the entire file into memory before sending anything back to the client. That's why ExtFile 1.4 is so much better - it uses RESPONSE.write() with 64k chunks. I just don't

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-08 Thread Paul Winkler
On Wed, Mar 24, 2004 at 12:57:00PM -0500, Chris McDonough wrote: moved to zope-dev only On Wed, 2004-03-24 at 09:28, Shane Hathaway wrote: This sounds useful for serving content from the filesystem. However, I'm a little concerned about this because producers must not read from the

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-08 Thread Paul Winkler
On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote: Jeremy has suggested that object pre-fetching could be added to ZODB. This is much on my mind currently. Any thoughts on what an API for pre-fetching might look like? The use case that most concerns me is: If you have an Image or

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-08 Thread Shane Hathaway
On 04/08/04 12:16, Paul Winkler wrote: On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote: Jeremy has suggested that object pre-fetching could be added to ZODB. This is much on my mind currently. Any thoughts on what an API for pre-fetching might look like? Well, thinking about it

RE: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-08 Thread zope
In fact, Zope puts large files (the threshold is around 256K - 512K) into a temporary file before serving them, to free up application threads. It's a tremendous handicap. I'm working on a product which serves files from the filesystem. The data retrieval method is the usual: def

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-08 Thread Chris McDonough
I'm pretty sure that the tempfile penalty is unrelated to the results Paul sees in his tests, at least for smaller files. If the content-length header indicates that the body of the item is smaller than 128k, it does not appear to be spooled from a tempfile at all. This also may be why there

[Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-03-24 Thread Shane Hathaway
Chris McDonough wrote: I have put a new proposal up at http://dev.zope.org/Wikis/DevSite/Proposals/FasterStaticContentServing which deals with serving large static content objects faster from Zope 2. This is based on some work that Paul Winkler and I did at the PyCon Zope 2 sprint. Comments

[Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-03-24 Thread Chris McDonough
moved to zope-dev only On Wed, 2004-03-24 at 09:28, Shane Hathaway wrote: This sounds useful for serving content from the filesystem. However, I'm a little concerned about this because producers must not read from the object database. I think the easiest thing to do is to turn this into a

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-03-24 Thread Shane Hathaway
Chris McDonough wrote: IMO code that needs to read from the database shouldn't return a producer. Instead, it should probably continue using the RESPONSE.write streaming protocol in the worker thread when it needs to do producer-like things. Returning a producer to ZPublisher seems to only

Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-03-24 Thread Chris McDonough
On Wed, 2004-03-24 at 13:32, Shane Hathaway wrote: Chris McDonough wrote: IMO code that needs to read from the database shouldn't return a producer. Instead, it should probably continue using the RESPONSE.write streaming protocol in the worker thread when it needs to do producer-like