Roché Compaan wrote at 2008-8-25 17:36 +0200:
>On Sun, 2008-08-24 at 08:55 +0200, Roché Compaan wrote:
>> Thanks for the feedback. I'll re-run the tests without any text indexes,
>> as well as run it with other implementations such as TextIndexNG3 and
>> SimpleTextIndex and compare the results.
>Some more tests show that text indexes aren't the worst offenders. Date
>and DateRangeIndexes use IISet in cases where IITreeSet seem more
>appropriate. To me there isn't much more value to investigate other text
>index implementations. I'd rather spend to time to compare the overall
>results with other indexing implementations altogether, like solr or
>indexing in a RDMBS.
>Listed below are some stats (where I ran my original test in which I
>create 10000 documents) that compare an unmodified setup, a catalog
>without text indexes, a catalog without date indexes, a catalog without
>metadata and no catalog at all.
>Total size of default setup:              2569.97 MB
>Total size excluding text indexes:        1963.89 MB

This means text indexes cost about 600 MB (25 %).

>Total size excluding date range indexes:  2043.26 MB

This means range indexes cost about 500 MB.

You may consider a "Managable RangeIndex" instead of the standard
range indexes.

With "Managable RangeIndex" a "DateRangeIndex" is implemented
as a "RangeIndex" with data type "DateInteger" or "DateTimeInteger".

If you also use "dm.incrementalsearch" with "Products.AdvancedQuery",
then you can replace the (expensive, both in terms of storage
as well as runtime) range indexes by incremental filtering --
which may not only let you save lots of space but also can
give dramatic speed improvements.

For more information about ZODB, see the ZODB Wiki:

ZODB-Dev mailing list  -

Reply via email to