On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote:
Jeremy has suggested that object pre-fetching could be added to ZODB.
This is much on my mind currently. Any thoughts on what an API for pre-fetching might look like?
Well, thinking about it some more, it seems like it might be just as easy to run a special thread just for fetching blobs. One queue should tell the blob thread what to get, and another queue should return the results. I think that would be a nice design.
The use case that most concerns me is: If you have an Image or File object with a very long Pdata chain, you're likely to A) thrash the ZEO client cache if it's not big enough, and B) spend a looooong time waiting for all the objects in the chain to load. At least, this is my theory on what's slowing things down - I will be trying to verify this today. See the bottom of this page: http://www.slinkp.com/code/zopestuff/blobnotes
In fact, Zope puts large files (the threshold is around 256K - 512K) into a temporary file before serving them, to free up application threads. It's a tremendous handicap.
Here's a relevant tangent. Twisted and asyncore like all I/O to be event-driven, but ZODB is fundamentally not event-driven. In ZODB, any attribute access can incur I/O time, and you can't control that. This is why Stackless is interesting for ZODB. With Stackless, you might be able to switch stack frames at the moment ZODB needs to do some I/O and get some useful work done. Without Stackless, ZODB can not guarantee any particular response time and therefore should not operate inside a time-critical event loop. Threads can also solve this problem, but the global interpreter lock hurts performance. There's also POSH. With POSH, you can take advantage of multiple processors (which you can't do with Stackless nor threads)... that seems like a really good thing. Some careful coding might make Zope + POSH scream.
Zope-Dev maillist - [EMAIL PROTECTED]
** No cross posts or HTML encoding! **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce