On 20 Feb 2014, at 2:27 am, Jim Fulton <j...@zope.com> wrote: > On Wed, Feb 19, 2014 at 9:57 AM, Dylan Jay <d...@pretaweb.com> wrote: >> On 19 Feb 2014, at 10:44 pm, Jim Fulton <j...@zope.com> wrote: > ... >> yeah I figured it might be the case thats its hard to predict. In this case >> it's catalog indexing so I was wondering if something could be done with >> __iter__ on a btree? It's a reasonably good guess that you could start >> preloading more of those objects if the first few are loaded? > > Iterators certainly seem like a logical place to start.
As an example I originally was doing a TTW zope reindex of a single index. Due to conflict problems I used a modified version of this https://github.com/plone/Products.PloneOrg/blob/master/scripts/catalog_rebuild.py (which I'd love to integrate something similar into zcatalog sometime). Both use iterators I believe. I think even if there was an explicit api where you can pass in an iterator, a max buffer length and you'd get passed back another iterator. Then asynchronously objects will load to try and keep ahead of the iterator consumption. e.g. for obj in async_load(myitr, 50): dox(obj) I don't know how that would help with a loop like this however for obj in async_load(myitr, 50): dox(obj.getMainObject()) > > Jim > > -- > Jim Fulton > http://www.linkedin.com/in/jimfulton _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
