On Sun, 2008-08-24 at 08:55 +0200, Roché Compaan wrote: > Thanks for the feedback. I'll re-run the tests without any text indexes, > as well as run it with other implementations such as TextIndexNG3 and > SimpleTextIndex and compare the results. >
Some more tests show that text indexes aren't the worst offenders. Date and DateRangeIndexes use IISet in cases where IITreeSet seem more appropriate. To me there isn't much more value to investigate other text index implementations. I'd rather spend to time to compare the overall results with other indexing implementations altogether, like solr or indexing in a RDMBS. Listed below are some stats (where I ran my original test in which I create 10000 documents) that compare an unmodified setup, a catalog without text indexes, a catalog without date indexes, a catalog without metadata and no catalog at all. Total size of default setup: 2569.97 MB Total size excluding text indexes: 1963.89 MB Total size excluding date range indexes: 2043.26 MB Total size excluding catalog: 270.90 MB Unmodified setup (default portal_catalog): ========================================== Classname, Count, Size (kB), Size (mB), Avg (kB) ------------------------------------------------ Products.PlonePAS.tools.memberdata.MemberData, 1, 0, 0, 0 OFS.Folder.Folder, 10000, 537, 0.52, 0.05 BTrees.Length.Length, 220107, 6382, 6.23, 0.03 Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25 Persistence.mapping.PersistentMapping, 20000, 8261, 8.07, 0.41 Products.ATContentTypes.content.document.ATDocument, 10000, 15077, 14.72, 1.51 BTrees._OIBTree.OIBTree, 4613, 23008, 22.47, 4.99 BTrees._IIBTree.IITreeSet, 49383, 25975, 25.37, 0.53 BTrees._OOBTree.OOBTree, 15875, 52354, 51.13, 3.3 BTrees._IIBTree.IIBTree, 143942, 53566, 52.31, 0.37 BTrees._OOBTree.OOBucket, 115332, 70789, 69.13, 0.61 BTrees._IOBTree.IOBTree, 25645, 71072, 69.41, 2.77 BTrees._OIBTree.OIBucket, 132417, 101472, 99.09, 0.77 BTrees._IIBTree.IIBucket, 252121, 163524, 159.69, 0.65 BTrees._IOBTree.IOBucket, 655025, 1007623, 984.01, 1.54 BTrees._IIBTree.IISet, 640686, 1024506, 1000.49, 1.6 Total size: 2569.97 MB No text indexes: ================ Classname, Count, Size (kB), Size (mB), Avg (kB) ------------------------------------------------ OFS.Folder.Folder, 10000, 537, 0.52, 0.05 BTrees.Length.Length, 120077, 3397, 3.32, 0.03 Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25 Persistence.mapping.PersistentMapping, 20000, 8242, 8.05, 0.41 Products.ATContentTypes.content.document.ATDocument, 10000, 15057, 14.7, 1.51 BTrees._OIBTree.OIBTree, 3467, 15740, 15.37, 4.54 BTrees._IIBTree.IITreeSet, 45734, 24355, 23.78, 0.53 BTrees._IOBTree.IOBTree, 20895, 37559, 36.68, 1.8 BTrees._OOBTree.OOBTree, 15878, 52460, 51.23, 3.3 BTrees._OOBTree.OOBucket, 115336, 70789, 69.13, 0.61 BTrees._OIBTree.OIBucket, 92499, 75572, 73.8, 0.82 BTrees._IOBTree.IOBucket, 287777, 670918, 655.19, 2.33 BTrees._IIBTree.IISet, 644224, 1028891, 1004.78, 1.6 Total size: 1963.89 MB No date and date range indexes: =============================== Classname, Count, Size (kB), Size (mB), Avg (kB) ------------------------------------------------ OFS.Folder.Folder, 10000, 537, 0.52, 0.05 BTrees.Length.Length, 220002, 6379, 6.23, 0.03 Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25 Persistence.mapping.PersistentMapping, 20000, 8242, 8.05, 0.41 Products.ATContentTypes.content.document.ATDocument, 10000, 15057, 14.7, 1.51 BTrees._OIBTree.OIBTree, 2688, 16786, 16.39, 6.24 BTrees._IIBTree.IITreeSet, 35868, 20593, 20.11, 0.57 BTrees._OOBTree.OOBTree, 15898, 52997, 51.75, 3.33 BTrees._IIBTree.IIBTree, 143806, 53443, 52.19, 0.37 BTrees._IOBTree.IOBTree, 24944, 68928, 67.31, 2.76 BTrees._OOBTree.OOBucket, 115356, 70806, 69.15, 0.61 BTrees._OIBTree.OIBucket, 81022, 76848, 75.05, 0.95 BTrees._IIBTree.IIBucket, 251985, 163777, 159.94, 0.65 BTrees._IIBTree.IISet, 582704, 519508, 507.33, 0.89 BTrees._IOBTree.IOBucket, 644640, 1010896, 987.2, 1.57 Total size: 2043.26 MB No metadata: ============ Classname, Count, Size (kB), Size (mB), Avg (kB) ------------------------------------------------ OFS.Folder.Folder, 10000, 537, 0.52, 0.05 BTrees.Length.Length, 220095, 6381, 6.23, 0.03 Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25 Persistence.mapping.PersistentMapping, 20000, 8242, 8.05, 0.41 Products.ATContentTypes.content.document.ATDocument, 10000, 15057, 14.7, 1.51 BTrees._OIBTree.OIBTree, 4608, 22971, 22.43, 4.99 BTrees._IIBTree.IITreeSet, 47718, 25185, 24.59, 0.53 BTrees._OOBTree.OOBTree, 15882, 52566, 51.33, 3.31 BTrees._IIBTree.IIBTree, 143948, 53520, 52.27, 0.37 BTrees._IOBTree.IOBTree, 25604, 69710, 68.08, 2.72 BTrees._OOBTree.OOBucket, 115340, 70774, 69.12, 0.61 BTrees._OIBTree.OIBucket, 132412, 101655, 99.27, 0.77 BTrees._IIBTree.IIBucket, 252127, 164492, 160.64, 0.65 BTrees._IOBTree.IOBucket, 655008, 689299, 673.14, 1.05 BTrees._IIBTree.IISet, 642280, 1027442, 1003.36, 1.6 Total size: 2261.07 MB No catalog: =========== Classname, Count, Size (kB), Size (mB), Avg (kB) ------------------------------------------------ OFS.Folder.Folder, 9524, 511, 0.5, 0.05 BTrees._IIBTree.IITreeSet, 550, 632, 0.62, 1.15 BTrees.Length.Length, 66668, 1904, 1.86, 0.03 BTrees._OIBTree.OIBTree, 1114, 6211, 6.07, 5.58 Products.Archetypes.BaseUnit.BaseUnit, 28572, 7147, 6.98, 0.25 Persistence.mapping.PersistentMapping, 19048, 7722, 7.54, 0.41 BTrees._IIBTree.IISet, 19122, 9239, 9.02, 0.48 BTrees._IOBTree.IOBTree, 3312, 10453, 10.21, 3.16 Products.ATContentTypes.content.document.ATDocument, 9524, 14352, 14.02, 1.51 BTrees._OOBTree.OOBTree, 12010, 23170, 22.63, 1.93 BTrees._OOBTree.OOBucket, 49857, 30574, 29.86, 0.61 BTrees._OIBTree.OIBucket, 29354, 33597, 32.81, 1.14 BTrees._IOBTree.IOBucket, 88074, 131886, 128.79, 1.5 Total size: 270.9 MB -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
