Hi Jim. This approach is completely new to me. It is interesting and
I'll keep an eye on your blog. How expensive is updating indexes with
data modification. You are talking about sqlite, does this mean you
figure the indexes would be a significant drag on the ZODB due their
size or is this idea a performance consideration. Many thanks.
Jim Washington wrote:
This is probably a bit premature, but I have been doing some thinking
about reordering/batching of large sets.
My current (not-quite-ready-for-prime-time) solution involves
factoradics, and I have done a bit of a write-up on my blog,
The idea is that you can have indexes to several permutations
(first-name-sort, last-name sort, last-modified-date-sort, etc) of a
set, and refer to those permutations by name, which refers to an integer
index. The factoradic representation of that integer can be used in a
quick algorithm to make a list of the ids in the set in that specific
order, and simply slicing that list would give easy batching capability
without needing to access the objects at all. There would, however, be
a need to update all of the permutation indexes when the data change.
One thing I have not dealt with yet to make this really useful would be
a utility that memoizes the factorials and factoradics, which are
expensive to calculate, probably using an sqlite database.
The big trade-off is doing a sort each time the data is accessed vs.
doing all of the named sorts each time the data is changed.
I am not sure at the moment whether this solves a real need yet, but I
thought I would bring it up since we are discussing batching. I think
it is just at the "interesting idea" stage for right now.
Zope3-users mailing list