On 2013-11-12 17:48, Martijn Pieters wrote:
On Mon, Nov 11, 2013 at 9:24 PM, Daniel Widerin <dan...@widerin.net
Anyone experienced similar problems packing large relstorage databases?
The graph traversal takes a really long time. maybe we can improve that
by storing additional information in the relational database?
You should (at least initially) pack *without* GC (set pack-gc true to
false); I packed a humongous RelStorage-backed database before, and
packed to earlier dates in the past first to minimize the amount of data
removed in a single transaction.
Only when we were down to a reasonable size database did we enable
Thanks Martijn for the hint, but we are using a history free database,
so growing does only happen by deleted objects in our case.
When in history free mode, is it possible to detect deleted objects at
store-time? This way we could add the zoid at store time to a
objects_deleted table in order to clean them up later.
Another way to speed up graph traversal would be to store the
object-references in a field of object_state. At the moment we have to
read the pickle in order to get the referenced zoids. Storing additional
- redundant - information might be not perfect, but it would allow to
pack/gc the database without any knowledge about the state objects
structure, i.e. using a stored procedure.
I would like to know what the relstorage experts think about this ideas.
Klein & Partner KG, member of BlueDynamics Alliance
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org