I'm pretty sure that the tempfile penalty is unrelated to the results Paul sees in his tests, at least for smaller files.
If the content-length header indicates that the body of the item is smaller than 128k, it does not appear to be spooled from a tempfile at all. This also may be why there isn't much of a difference between "normal" product results and Paul's "patched" product results at his 12K file size as opposed to the roughly 60% difference in speed at at 324K and 1216K (I cannot explain the large difference in results at 38322k across products; it appears to be product-specific). Also, for this file size, wrapping the data in a producer is actually just more overhead; products would be better off to do a direct RESPONSE.write if the total file size was under 128k (as evidenced by the fact that the iterator-based products are slower at small file sizes). It's a bit of a mystery about why retrieving data over ZEO is an order of magnitude slower than retrieving it from a local file storage when the ZEO server is actually on the same physical machine. I'm sure there are lots of optimization strategies for making this faster that involve a lot of heavy lifting and hard thinking in a dark room. ;-) While that might be fun, in the meantime, for the hoi-polloi who just need to get stuff done, I believe that one simple answer to this is, as usual, to do very aggressive caching and avoid ZODB access altogether for large blobs. The ZEO caching algorithm is largely agnostic about what kinds of objects it caches and doesn't allow for setting a cache-eviction policy other than whatever its implementors consider optimal for the most general case. This would be fairly difficult to change in the best-case circumstance for someone with deep ZODB knowledge. A more effective strategy for "normal" people might be to contribute to the "hardening" of something like FileCacheManager, which prevents ZODB access in the first place and allows for fine-grained policies on an application level by caching blobs on client filesystems. - C On Thu, 2004-04-08 at 12:16, Paul Winkler wrote: > On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote: > > Jeremy has suggested that object pre-fetching could be added to ZODB. > > This is much on my mind currently. > Any thoughts on what an API for pre-fetching might look like? > > The use case that most concerns me is: > If you have an Image or File object with a very long Pdata chain, > you're likely to A) thrash the ZEO client cache if it's not big enough, > and B) spend a looooong time waiting for all the objects in the chain > to load. At least, this is my theory on what's slowing things > down - I will be trying to verify this today. See the bottom of this page: > http://www.slinkp.com/code/zopestuff/blobnotes _______________________________________________ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
