Chris McDonough wrote:
IMO code that needs to read from the database shouldn't return a
producer.  Instead, it should probably continue using the RESPONSE.write
streaming protocol in the worker thread when it needs to do
producer-like things.  Returning a producer to ZPublisher seems to only
really useful when the producer's "more" generator is guaranteed to be
exceedingly cheap because as you noted it is meant to operate in the
context of the main thread.


I'll note that iterators probably ought to replace producers. Just spell "more" as "next" and they look pretty much the same.


The time spent waiting for the code that accessed the database would
block all other asyncore operations, though, right?  We'd need to test
it, but I suspect it might be a net lose for the "multiple requests for
the same object" case because the overhead of reading from the database
cache would be realized serially for each request.

Look at it this way:


- Don't ghostify anything manually. Let ZODB handle that.

- Use a larger ZODB cache for the main thread's connection than you do for the other connections, to increase the chance that objects will be served directly from RAM.

- As long as other threads aren't reading/writing the large objects, there will be at most one copy of a large object in memory at any given time.

- Periodically ask the connection to collect garbage. It uses a LRU strategy, which seems much more optimal than immediate deactivation.

 And if the object
isn't in cache, it could potentially block for quite a long time.
That said, I dunno.  Do you think it might be a win?  I guess my worry
is that the the operation of producer should be  more or less
"guaranteed" to be cheap and it seems hard to make that promise about
ZODB access, especially as the data might be coming over the wire from
ZEO.

If the object is not loaded and not in the ZEO cache, the producer could say it's not ready yet and ask ZEO to fetch it in the background. Jeremy has suggested that object pre-fetching could be added to ZODB.


FWIW, Jim intimated a while back that he might be interested in
providing "blob" support directly within ZODB. I can imagine an
implementation of this where maybe you can mark an object as
"blobifiable" and when you do so, the ZODB caching code writes a copy of
that object into a named file on disk during normal operations
<hand-waving goes here ;->  Then we could use a producer to spool the
file data out without ever actually reading data out of a database from
a ZODB connection; we'd just ask the connection for the filename.

That's a possibility, although it would complicate the storage, and making it work with ZEO would require a distributed filesystem.


Shane

_______________________________________________
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )

Reply via email to