Re: [ZODB-Dev] Relstorage and over growing database.

[Daniel Widerin]

Hi, just want to share our experience:

My ZODB contains 300mio objects on relstorage/pgsql. The amount of
objects is caused by btrees stored on plone dexterity contenttypes. It's
size is 160GB. At that size it's impossible to pack because the pre-pack
takes 100 days.

jensens and me are searching for different packing algorithms and
methods to achieve better packing performance. We're keeping you updated
here!

How i solved my problem for now:

I converted into FileStorage which took about 40 hours and Data.fs was
55GB in size. Now i tried to run zeopack on that database - which
succeeded and database was reduced to 7.8 GB - still containing 40mio
objects. After that i migrated back to relstorage because of better
performance and the result is a 11 GB db in pgsql.

Anyone experienced similar problems packing large relstorage databases?
The graph traversal takes a really long time. maybe we can improve that
by storing additional information in the relational database?

Any hints or comments are welcome.

Daniel

Am 07.02.13 23:10, schrieb Shane Hathaway:
 On 02/07/2013 01:54 PM, Jürgen Herrmann wrote:
 Am 07.02.2013 21:18, schrieb Jürgen Herrmann:
 I know that's entirely not your fault but may be worth mentioning
 in the docs. Relstorage with MySQL works *very* well for DB sizes
 5GB or so, above that - not so much :/

 Also for the docs: on disk Restorage/MySQL uses 4x the size of a
 FileStorage with same contents. As packing tables are filled this
 grows by another factor of ~2. If you don't pack very regularly
 you might up ending in DBs that donb't permit packing anymore
 because of the big size very quickly.
 
 I suspect there are ways to fix all of that in the MySQL configuration.
 Like any SQL database, MySQL needs tuning as it grows.  Meanwhile,
 FileStorage doesn't really have any knobs, and it always stores in a
 fairly optimal way, so it's easier to use.
 
 FileStorage has a couple of issues that often drive people to
 RelStorage: (1) the on-disk format is unique to FileStorage, and there
 aren't many tools available for analyzing and fixing a broken Data.fs.
 (2) FileStorage only supports multiple clients through ZEO, which has
 relatively high latency.  If these issues don't impact you, then
 FileStorage is clearly the better choice for you.
 
 Shane
 
 ___
 For more information about ZODB, see http://zodb.org/
 
 ZODB-Dev mailing list  -  ZODB-Dev@zope.org
 https://mail.zope.org/mailman/listinfo/zodb-dev


___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] ZODB KeyError

[Daniel Widerin]

Hi all,

I'm getting a ZODB KeyError on every write access to the ZODB. Reading 
is possible without problems. This ZODB is 2.4 GB at size an running a 
single Plone 3.3.3 site, ZODB Version is 3.8.5 *without* plone.app.blob 
installed. I used 3.8.x because i have an seperated catalog mountpoint 
using it's own storage and need cross-reference-packing support.


Traceback (innermost last):

 * Module ZPublisher.Publish, line 119, in publish
 * Module ZPublisher.mapply, line 88, in mapply
 * Module ZPublisher.Publish, line 42, in call_object
 * Module Products.CMFPlone.FactoryTool, line 371, in __call__
 * Module Products.CMFPlone.FactoryTool, line 155, in __getitem__
 * Module Products.CMFPlone.PloneFolder, line 348, in invokeFactory
 * Module Products.CMFCore.TypesTool, line 716, in constructContent
 * Module Products.CMFCore.TypesTool, line 276, in constructInstance
 * Module Products.CMFCore.TypesTool, line 450, in _constructInstance
 * Module Products.ATContentTypes.content.image, line 4, in addATImage
 * Module OFS.ObjectManager, line 348, in _setObject
 * Module zope.event, line 23, in notify
 * Module zope.component.event, line 26, in dispatch
 * Module zope.component._api, line 131, in subscribers
 * Module zope.component.registry, line 300, in subscribers
 * Module zope.interface.adapter, line 575, in subscribers
 * Module zope.component.event, line 33, in objectEventNotify
 * Module zope.component._api, line 131, in subscribers
 * Module zope.component.registry, line 300, in subscribers
 * Module zope.interface.adapter, line 575, in subscribers
 * Module five.intid.intid, line 81, in addIntIdSubscriber
 * Module five.intid.intid, line 36, in register
 * Module zope.app.intid, line 106, in _generateId
 * Module ZODB.Connection, line 815, in setstate
 * Module ZODB.Connection, line 874, in _setstate
 * Module ZODB.serialize, line 604, in setGhostState
 * Module ZODB.serialize, line 597, in getState
 * Module ZODB.serialize, line 479, in _persistent_load
 * Module ZODB.serialize, line 543, in load_multi_oid
 * Module ZODB.serialize, line 537, in load_oid
 * Module ZODB.Connection, line 244, in get
 * Module ZEO.ClientStorage, line 712, in load
 * Module ZEO.ClientStorage, line 735, in loadEx
 * Module ZEO.ServerStub, line 196, in loadEx
 * Module ZEO.zrpc.connection, line 699, in call

KeyError: '\x00\x00\x00\x00\x00\x01#\xc6'



Any hints?
btw: ZEO and ZOPE Clients run on different (virtual) servers. Python 
Version is 2.4.6, Zope Version is 2.10.9, OS is Debian 5.0

Thanks

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev