Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jan-Wijbrand Kolman
On 11/16/13 2:01 AM, Jens W. Klein wrote: Did I miss something? Any opinions much appreciated! Expect updates in this thread :) We did experience relstorage problems that we think could have been related to packing - we're not sure yet. So, I'm following this thread and your updates with

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jim Fulton
On Fri, Nov 15, 2013 at 8:01 PM, Jens W. Klein j...@bluedynamics.com wrote: I started a new packing script for Relstorage (history free, postgresql). It is based on incoming reference counting. Did you look at zc.zodbdgc? I think it implements something very close to what you're proposing.

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jan-Wijbrand Kolman
On 11/18/13 12:19 PM, Jim Fulton wrote: On Fri, Nov 15, 2013 at 8:01 PM, Jens W. Klein j...@bluedynamics.com wrote: I started a new packing script for Relstorage (history free, postgresql). It is based on incoming reference counting. Did you look at zc.zodbdgc? I think it implements

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jim Fulton
On Mon, Nov 18, 2013 at 8:43 AM, Jan-Wijbrand Kolman janwijbr...@gmail.com wrote: On 11/18/13 12:19 PM, Jim Fulton wrote: On Fri, Nov 15, 2013 at 8:01 PM, Jens W. Klein j...@bluedynamics.com wrote: I started a new packing script for Relstorage (history free, postgresql). It is based on

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Shane Hathaway
On 11/15/2013 06:01 PM, Jens W. Klein wrote: The idea is simple: - iterate over all transactions starting with the lowest transaction id (tid) - for each transaction load the object states connected with tid - for each state fetch its outgoing references and fill a table where all

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jens W. Klein
Hi Jim, thanks for the hint (also in the other post). I looked at zc.zodbdgc and took some inspiration from it. As far as I understand it stores the incoming references in a separate filestorage backend. So this works similar to my impelmentation but uses the ZODB infrastructure. I dont see

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jens W. Klein
On 2013-11-18 17:29, Shane Hathaway wrote: On 11/15/2013 06:01 PM, Jens W. Klein wrote: The idea is simple: - iterate over all transactions starting with the lowest transaction id (tid) - for each transaction load the object states connected with tid - for each state fetch its outgoing

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jim Fulton
On Mon, Nov 18, 2013 at 12:00 PM, Jens W. Klein j...@bluedynamics.com wrote: Hi Jim, thanks for the hint (also in the other post). I looked at zc.zodbdgc and took some inspiration from it. As far as I understand it stores the incoming references in a separate filestorage backend. This is