On 2013-11-12 17:48, Martijn Pieters wrote:
On Mon, Nov 11, 2013 at 9:24 PM, Daniel Widerin <dan...@widerin.net <mailto:dan...@widerin.net>> wrote: Anyone experienced similar problems packing large relstorage databases? The graph traversal takes a really long time. maybe we can improve that by storing additional information in the relational database? You should (at least initially) pack *without* GC (set pack-gc true to false); I packed a humongous RelStorage-backed database before, and packed to earlier dates in the past first to minimize the amount of data removed in a single transaction. Only when we were down to a reasonable size database did we enable garbage collection.
Thanks Martijn for the hint, but we are using a history free database, so growing does only happen by deleted objects in our case.
When in history free mode, is it possible to detect deleted objects at store-time? This way we could add the zoid at store time to a objects_deleted table in order to clean them up later.
Another way to speed up graph traversal would be to store the object-references in a field of object_state. At the moment we have to read the pickle in order to get the referenced zoids. Storing additional - redundant - information might be not perfect, but it would allow to pack/gc the database without any knowledge about the state objects structure, i.e. using a stored procedure.
I would like to know what the relstorage experts think about this ideas. kind regards Jens -- Klein & Partner KG, member of BlueDynamics Alliance _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
