Roché Compaan wrote at 2008-8-25 17:36 +0200: >On Sun, 2008-08-24 at 08:55 +0200, Roché Compaan wrote: >> Thanks for the feedback. I'll re-run the tests without any text indexes, >> as well as run it with other implementations such as TextIndexNG3 and >> SimpleTextIndex and compare the results. >> > >Some more tests show that text indexes aren't the worst offenders. Date >and DateRangeIndexes use IISet in cases where IITreeSet seem more >appropriate. To me there isn't much more value to investigate other text >index implementations. I'd rather spend to time to compare the overall >results with other indexing implementations altogether, like solr or >indexing in a RDMBS. > >Listed below are some stats (where I ran my original test in which I >create 10000 documents) that compare an unmodified setup, a catalog >without text indexes, a catalog without date indexes, a catalog without >metadata and no catalog at all. > >Total size of default setup: 2569.97 MB >Total size excluding text indexes: 1963.89 MB
This means text indexes cost about 600 MB (25 %). >Total size excluding date range indexes: 2043.26 MB This means range indexes cost about 500 MB. You may consider a "Managable RangeIndex" instead of the standard range indexes. With "Managable RangeIndex" a "DateRangeIndex" is implemented as a "RangeIndex" with data type "DateInteger" or "DateTimeInteger". If you also use "dm.incrementalsearch" with "Products.AdvancedQuery", then you can replace the (expensive, both in terms of storage as well as runtime) range indexes by incremental filtering -- which may not only let you save lots of space but also can give dramatic speed improvements. -- Dieter _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
