On 2013-11-12 17:48, Martijn Pieters wrote:

On Mon, Nov 11, 2013 at 9:24 PM, Daniel Widerin <dan...@widerin.net
<mailto:dan...@widerin.net>> wrote:

    Anyone experienced similar problems packing large relstorage databases?
    The graph traversal takes a really long time. maybe we can improve that
    by storing additional information in the relational database?

You should (at least initially) pack *without* GC (set pack-gc true to
false); I packed a humongous RelStorage-backed database before, and
packed to earlier dates in the past first to minimize the amount of data
removed in a single transaction.

Only when we were down to a reasonable size database did we enable
garbage collection.

Thanks Martijn for the hint, but we are using a history free database, so growing does only happen by deleted objects in our case.

When in history free mode, is it possible to detect deleted objects at store-time? This way we could add the zoid at store time to a objects_deleted table in order to clean them up later.

Another way to speed up graph traversal would be to store the object-references in a field of object_state. At the moment we have to read the pickle in order to get the referenced zoids. Storing additional - redundant - information might be not perfect, but it would allow to pack/gc the database without any knowledge about the state objects structure, i.e. using a stored procedure.

I would like to know what the relstorage experts think about this ideas.

kind regards
Klein & Partner KG, member of BlueDynamics Alliance

For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org

Reply via email to