Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-09 Thread Paul Winkler
On Fri, Apr 09, 2004 at 12:25:50AM -0400, Chris McDonough wrote:
 and 1216K (I cannot explain the large difference in results at 38322k
 across products; it appears to be product-specific).

That's easy. LocalFS, CMFCore.FSFile, and ExtFile 1.1.3 all read the
entire file into memory before sending anything back to the client.
That's why ExtFile 1.4 is so much better - it uses RESPONSE.write()
with 64k chunks.

  Also, for this
 file size, wrapping the data in a producer is actually just more

iterators now, remember? :-)

 overhead; products would be better off to do a direct RESPONSE.write if
 the total file size was under 128k (as evidenced by the fact that the
 iterator-based products are slower at small file sizes).

Yeah, my patches were very minimal. They could easily be extended to
only use iterators for larger content-length.
I'll do that, add a working LocalFS and another thingie I heard about 
yesterday into the mix, and do another round of benchmarks maybe tomorrow.

 It's a bit of a mystery about why retrieving data over ZEO is an order
 of magnitude slower than retrieving it from a local file storage when
 the ZEO server is actually on the same physical machine.   I'm sure
 there are lots of optimization strategies for making this faster that
 involve a lot of heavy lifting and hard thinking in a dark room. ;-) 
 While that might be fun, in the meantime, for the hoi-polloi who just
 need to get stuff done, I believe that one simple answer to this is, as
 usual, to do very aggressive caching and avoid ZODB access altogether
 for large blobs.  The ZEO caching algorithm is largely agnostic about
 what kinds of objects it caches and doesn't allow for setting a
 cache-eviction policy other than whatever its implementors consider
 optimal for the most general case.   This would be fairly difficult to
 change in the best-case circumstance for someone with deep ZODB
 knowledge.  A more effective strategy for normal people might be to
 contribute to the hardening of something like FileCacheManager, which
 prevents ZODB access in the first place and allows for fine-grained
 policies on an application level by caching blobs on client filesystems.

Fine and dandy, but I'd still really love to see what can be done
about ZEO. Caches only prevent ZODB access for stuff that's in the cache...
Sooner or later, some user will get a cache miss and that poor soul will 
see speed go down by an order of magnitude or two.  Ouch.

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's HIGHLY TOXIC PENGUIN THING!
(random hero from isometric.spaceninja.com)

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

2004-04-09 Thread Chris McDonough
On Fri, 2004-04-09 at 18:02, Paul Winkler wrote:
 That's easy. LocalFS, CMFCore.FSFile, and ExtFile 1.1.3 all read the
 entire file into memory before sending anything back to the client.
 That's why ExtFile 1.4 is so much better - it uses RESPONSE.write()
 with 64k chunks.

I just don't understand why it's a third the speed of ExtFile 1.1.3 at
the largest file size.


  overhead; products would be better off to do a direct RESPONSE.write if
  the total file size was under 128k (as evidenced by the fact that the
  iterator-based products are slower at small file sizes).
 
 Yeah, my patches were very minimal. They could easily be extended to
 only use iterators for larger content-length.
 I'll do that, add a working LocalFS and another thingie I heard about 
 yesterday into the mix, and do another round of benchmarks maybe tomorrow.

Cool...


 Fine and dandy, but I'd still really love to see what can be done
 about ZEO. Caches only prevent ZODB access for stuff that's in the cache...
 Sooner or later, some user will get a cache miss and that poor soul will 
 see speed go down by an order of magnitude or two.  Ouch.

Since it's only on the first request, I don't care *too* much (although
I do care) how long it takes to slurp across ZEO, especially because
blobs rarely change in normal usage and thus would be seldom evicted
from FSCacheManager's cache.  It's just likely more bang for the buck
in the common case to prevent the ZODB read in the first place than to
try to pick through and speed up ZEO's (very subtle) code.  That said,
go for it!

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )