On 20 Feb 2014, at 2:27 am, Jim Fulton <j...@zope.com> wrote:

> On Wed, Feb 19, 2014 at 9:57 AM, Dylan Jay <d...@pretaweb.com> wrote:
>> On 19 Feb 2014, at 10:44 pm, Jim Fulton <j...@zope.com> wrote:
> ...
>> yeah I figured it might be the case thats its hard to predict. In this case 
>> it's catalog indexing so I was wondering if something could be done with 
>> __iter__ on a btree? It's a reasonably good guess that you could start 
>> preloading more of those objects if the first few are loaded?
> 
> Iterators certainly seem like a logical place to start.

As an example I originally was doing a TTW zope reindex of a single index.
Due to conflict problems I used a modified version of this 
https://github.com/plone/Products.PloneOrg/blob/master/scripts/catalog_rebuild.py
 (which I'd love to integrate something similar into zcatalog sometime).
Both use iterators I believe.
I think even if there was an explicit api where you can pass in an iterator, a 
max buffer length and you'd get passed back another iterator. Then 
asynchronously objects will load to try and keep ahead of the iterator 
consumption. 
e.g.
for obj in async_load(myitr, 50):
   dox(obj)

I don't know how that would help with a loop like this however

for obj in async_load(myitr, 50):
   dox(obj.getMainObject())





> 
> Jim
> 
> -- 
> Jim Fulton
> http://www.linkedin.com/in/jimfulton

_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Reply via email to