Re: [ZODB-Dev] IStorageIteration
Hi, On Feb 11, 2008, at 4:19 PM, Christian Theune wrote: Context could be given as a reference date that is opaque to the client and can differ from storage to storage, a file pointer could serve this purpose. The API still might include the `length` of the data returned to minimize round-trips. Actually, a file position doesn't work either, because the storage server can't know that much about what it is serving. I would have thought that this wouldn't matter if we declare the context `opaque`. As long as the storage can hand out a token that represents context and that can be transferred through zrpc ... I suspect the only sane approach is to make the storage server maintain the iterator returned by the underlying storage on behalf of the client. This will cause resources to be allocated on the server that might only be freed when the client disconnects. Right, now this makes me feel uneasy. :) Christian ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] IStorageIteration
On Feb 11, 2008, at 4:19 PM, Christian Theune wrote: Context could be given as a reference date that is opaque to the client and can differ from storage to storage, a file pointer could serve this purpose. The API still might include the `length` of the data returned to minimize round-trips. Actually, a file position doesn't work either, because the storage server can't know that much about what it is serving. I suspect the only sane approach is to make the storage server maintain the iterator returned by the underlying storage on behalf of the client. This will cause resources to be allocated on the server that might only be freed when the client disconnects. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] IStorageIteration
On Feb 11, 2008, at 8:58 AM, Christian Theune wrote: Hi, the interface IStorageIteration says: class IStorageIteration(Interface): API for iterating over the contents of a storage Note that this is a future API. Some storages now provide an approximation of this. [...] ZODB.FileStorage.FileStorage is the only class in the current ZODB code that implements this. I'm a little surprised. Also, this code isn't ZEO-compatible (apart from the fact that ClientStorage doesn't implement the interface). How so? Are there any plans to complete this feature? I would like to see this widely implemented. It is extremely useful. In practice, even when using ZEO, it is easy to use the file storage behind the ZEO. Note, though, that it is more useful to use ZODB.FileStorage.FileStorge.FileIterator directly as it doesn't require creating a a file-storage instance. (BTW, the module arrangement is such that you can't access the ZODB.FileStorage.FileStorge module directly as it gets replaced by the class. You have to dig it out of sys.module. :( ) We'd like to be able to iterate over transactions and their records through ZEO and we'd be happy to contribute (in the next days) so this works. +1 Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] IStorageIteration
Jim Fulton schrieb: On Feb 11, 2008, at 1:25 PM, Christian Theune wrote: Hi, Jim Fulton schrieb: On Feb 11, 2008, at 1:19 PM, Christian Theune wrote: ... Also, this code isn't ZEO-compatible (apart from the fact that ClientStorage doesn't implement the interface). How so? As far as I understand, ZEO can't pass iterators around directly but needs to provide at least some wrapping code on the protocol level ... Good point. This, at least, makes the implementation rather tricky. There is the pattern like undoInfo/Log and record_iternext which provides context by passing in ranges of records to return + tolerance for ranges that don't exist. I don't think that's going to work here. Iterating through the transactions in the database for each iteration is going to be totally non-scalable. Yes, you're right. Still, the road of providing call context via a parameter seems better to me than to enhance the ZEO protocol to account for this purpose. I suppose you could include file positions in the iterator state, although that makes me a little uncomfortable. It might be OK here though. Context could be given as a reference date that is opaque to the client and can differ from storage to storage, a file pointer could serve this purpose. The API still might include the `length` of the data returned to minimize round-trips. Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] IStorageIteration
On Feb 11, 2008, at 1:19 PM, Christian Theune wrote: ... Also, this code isn't ZEO-compatible (apart from the fact that ClientStorage doesn't implement the interface). How so? As far as I understand, ZEO can't pass iterators around directly but needs to provide at least some wrapping code on the protocol level ... Good point. This, at least, makes the implementation rather tricky. ... +1 Ok, cool. We'll try to figure out what needs to be done. And keep the list updated. OK Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
