Hi, just want to share our experience:
My ZODB contains 300mio objects on relstorage/pgsql. The amount of
objects is caused by btrees stored on plone dexterity contenttypes. It's
size is 160GB. At that size it's impossible to pack because the pre-pack
takes 100 days.
jensens and me are searching for different packing algorithms and
methods to achieve better packing performance. We're keeping you updated
here!
How i solved my problem for now:
I converted into FileStorage which took about 40 hours and Data.fs was
55GB in size. Now i tried to run zeopack on that database - which
succeeded and database was reduced to 7.8 GB - still containing 40mio
objects. After that i migrated back to relstorage because of better
performance and the result is a 11 GB db in pgsql.
Anyone experienced similar problems packing large relstorage databases?
The graph traversal takes a really long time. maybe we can improve that
by storing additional information in the relational database?
Any hints or comments are welcome.
Daniel
Am 07.02.13 23:10, schrieb Shane Hathaway:
On 02/07/2013 01:54 PM, Jürgen Herrmann wrote:
Am 07.02.2013 21:18, schrieb Jürgen Herrmann:
I know that's entirely not your fault but may be worth mentioning
in the docs. Relstorage with MySQL works *very* well for DB sizes
5GB or so, above that - not so much :/
Also for the docs: on disk Restorage/MySQL uses 4x the size of a
FileStorage with same contents. As packing tables are filled this
grows by another factor of ~2. If you don't pack very regularly
you might up ending in DBs that donb't permit packing anymore
because of the big size very quickly.
I suspect there are ways to fix all of that in the MySQL configuration.
Like any SQL database, MySQL needs tuning as it grows. Meanwhile,
FileStorage doesn't really have any knobs, and it always stores in a
fairly optimal way, so it's easier to use.
FileStorage has a couple of issues that often drive people to
RelStorage: (1) the on-disk format is unique to FileStorage, and there
aren't many tools available for analyzing and fixing a broken Data.fs.
(2) FileStorage only supports multiple clients through ZEO, which has
relatively high latency. If these issues don't impact you, then
FileStorage is clearly the better choice for you.
Shane
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev