On Fri, Apr 09, 2004 at 12:25:50AM -0400, Chris McDonough wrote:
and 1216K (I cannot explain the large difference in results at 38322k
across products; it appears to be product-specific).
That's easy. LocalFS, CMFCore.FSFile, and ExtFile 1.1.3 all read the
entire file into memory before sending anything back to the client.
That's why ExtFile 1.4 is so much better - it uses RESPONSE.write()
with 64k chunks.
Also, for this
file size, wrapping the data in a producer is actually just more
iterators now, remember? :-)
overhead; products would be better off to do a direct RESPONSE.write if
the total file size was under 128k (as evidenced by the fact that the
iterator-based products are slower at small file sizes).
Yeah, my patches were very minimal. They could easily be extended to
only use iterators for larger content-length.
I'll do that, add a working LocalFS and another thingie I heard about
yesterday into the mix, and do another round of benchmarks maybe tomorrow.
It's a bit of a mystery about why retrieving data over ZEO is an order
of magnitude slower than retrieving it from a local file storage when
the ZEO server is actually on the same physical machine. I'm sure
there are lots of optimization strategies for making this faster that
involve a lot of heavy lifting and hard thinking in a dark room. ;-)
While that might be fun, in the meantime, for the hoi-polloi who just
need to get stuff done, I believe that one simple answer to this is, as
usual, to do very aggressive caching and avoid ZODB access altogether
for large blobs. The ZEO caching algorithm is largely agnostic about
what kinds of objects it caches and doesn't allow for setting a
cache-eviction policy other than whatever its implementors consider
optimal for the most general case. This would be fairly difficult to
change in the best-case circumstance for someone with deep ZODB
knowledge. A more effective strategy for normal people might be to
contribute to the hardening of something like FileCacheManager, which
prevents ZODB access in the first place and allows for fine-grained
policies on an application level by caching blobs on client filesystems.
Fine and dandy, but I'd still really love to see what can be done
about ZEO. Caches only prevent ZODB access for stuff that's in the cache...
Sooner or later, some user will get a cache miss and that poor soul will
see speed go down by an order of magnitude or two. Ouch.
--
Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's HIGHLY TOXIC PENGUIN THING!
(random hero from isometric.spaceninja.com)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
