Re: [ZODB-Dev] grab zodb dict keys by search term

2013-12-11 Thread Jens W. Klein
you might be interested in https://pypi.python.org/pypi/souper/ which addresses this kind of problem hth Jens On 2013-12-10 23:46, Tamer Higazi wrote: Hi people! I am working a lot with IOB and OOB Trees as well with PersistentDict and PersistentList to store my Data in ZODB. I want to ask if

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jens W. Klein
how I make zc.zodbdgc play with Relstorage and since it works on the abstracted ZODB level using pickles I suspected it to be not fast enough for so many obejcts - so I skipped this alternative. Jens On 2013-11-18 12:19, Jim Fulton wrote: On Fri, Nov 15, 2013 at 8:01 PM, Jens W. Klein j

Re: [ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-18 Thread Jens W. Klein
On 2013-11-18 17:29, Shane Hathaway wrote: On 11/15/2013 06:01 PM, Jens W. Klein wrote: The idea is simple: - iterate over all transactions starting with the lowest transaction id (tid) - for each transaction load the object states connected with tid - for each state fetch its outgoing

[ZODB-Dev] Optimizing RelStorage Packing for large DBs

2013-11-15 Thread Jens W. Klein
I started a new packing script for Relstorage (history free, postgresql). It is based on incoming reference counting. Why do we need this? First a bit story telling (fast forward down to section new script if you're bored) The Problem --- So we had a database with ~298.000.000

Re: [ZODB-Dev] Relstorage and over growing database.

2013-11-13 Thread Jens W. Klein
On 2013-11-12 17:48, Martijn Pieters wrote: On Mon, Nov 11, 2013 at 9:24 PM, Daniel Widerin dan...@widerin.net mailto:dan...@widerin.net wrote: Anyone experienced similar problems packing large relstorage databases? The graph traversal takes a really long time. maybe we can improve