Hi, just want to share our experience: My ZODB contains 300mio objects on relstorage/pgsql. The amount of objects is caused by btrees stored on plone dexterity contenttypes. It's size is 160GB. At that size it's impossible to pack because the pre-pack takes >100 days.
jensens and me are searching for different packing algorithms and methods to achieve better packing performance. We're keeping you updated here! How i solved my problem for now: I converted into FileStorage which took about 40 hours and Data.fs was 55GB in size. Now i tried to run zeopack on that database - which succeeded and database was reduced to 7.8 GB - still containing 40mio objects. After that i migrated back to relstorage because of better performance and the result is a 11 GB db in pgsql. Anyone experienced similar problems packing large relstorage databases? The graph traversal takes a really long time. maybe we can improve that by storing additional information in the relational database? Any hints or comments are welcome. Daniel Am 07.02.13 23:10, schrieb Shane Hathaway: > On 02/07/2013 01:54 PM, Jürgen Herrmann wrote: >> Am 07.02.2013 21:18, schrieb Jürgen Herrmann: >>> I know that's entirely not your fault but may be worth mentioning >>> in the docs. Relstorage with MySQL works *very* well for DB sizes >>> <5GB or so, above that - not so much :/ >> >> Also for the docs: on disk Restorage/MySQL uses 4x the size of a >> FileStorage with same contents. As packing tables are filled this >> grows by another factor of ~2. If you don't pack very regularly >> you might up ending in DBs that donb't permit packing anymore >> because of the big size very quickly. > > I suspect there are ways to fix all of that in the MySQL configuration. > Like any SQL database, MySQL needs tuning as it grows. Meanwhile, > FileStorage doesn't really have any knobs, and it always stores in a > fairly optimal way, so it's easier to use. > > FileStorage has a couple of issues that often drive people to > RelStorage: (1) the on-disk format is unique to FileStorage, and there > aren't many tools available for analyzing and fixing a broken Data.fs. > (2) FileStorage only supports multiple clients through ZEO, which has > relatively high latency. If these issues don't impact you, then > FileStorage is clearly the better choice for you. > > Shane > > _______________________________________________ > For more information about ZODB, see http://zodb.org/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
