Re: [ZODB-Dev] savepoint not doing cacheGC?
On Thu, Mar 31, 2011 at 11:36 AM, Adam GROSZER wrote: > Hello, > > I'm trying to iterate over the IntIds utility's object in a huge > FileStorage DB, calling > transaction.savepoint(optimistic=True) > every 1000 iterations or so, but memory usage simply skyrockets. > Weird is that almost the same code was working with python 2.5 and ZODB > 3.9.6, now it does not with python 2.6 and ZODB 3.9.6. I don't think savepoints ever did any cacheGC calls. It's up to you to do that if you want. Savepoints only allow removal of modified objects from memory (as they're saved in temporary storage). If you aren't writing data, savepoints don't buy you anything and you should simply call cacheGC yourself periodically. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] savepoint not doing cacheGC?
Hello, I'm trying to iterate over the IntIds utility's object in a huge FileStorage DB, calling transaction.savepoint(optimistic=True) every 1000 iterations or so, but memory usage simply skyrockets. Weird is that almost the same code was working with python 2.5 and ZODB 3.9.6, now it does not with python 2.6 and ZODB 3.9.6. config is: cache-size 1 cache-size-bytes 50MB -- Best regards, Adam GROSZER -- Quote of the day: Patience is the companion of wisdom. - St. Augustine ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packing ZODB
Hello, On Thu, 31 Mar 2011 06:33:30 -0600 you wrote: > > On 03/31/2011 05:25 AM, Hanno Schlichting wrote: >> Hi. >> >> On Thu, Mar 31, 2011 at 12:46 PM, Adam GROSZER wrote: >>> After investigating FileStorage a bit, I found that GC runs on objects, >>> but pack later by transactions. That means that if there's a bigger-ish >>> transaction, we can't get rid of it until all of it's objects are GCed >>> (or superseeded by newer states). >>> >>> Is that correct? >> >> I think so. > > Actually we can remove pieces of the old transaction and that's the way > both FileStorage and RelStorage already work. RelStorage owes much of > its success to simply duplicating the functionality of FileStorage and > ZEO. ;-) Yup, checked that and it works... You just got to get rid of all references ;-) -- Best regards, Adam GROSZER -- Quote of the day: The proud man counts his newspaper clippings- the humble man his blessings. - Bishop Fulton J. Sheen ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packing ZODB
On 03/31/2011 04:46 AM, Adam GROSZER wrote: > After investigating FileStorage a bit, I found that GC runs on objects, > but pack later by transactions. That means that if there's a bigger-ish > transaction, we can't get rid of it until all of it's objects are GCed > (or superseeded by newer states). No, but we can remove old object states from the big transaction and mark it as partially destroyed (that's what the "packed" flag is for) so that the transaction is no longer eligible for undo. Both FileStorage and RelStorage do that. Not much space is wasted. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packing ZODB
Hi. On Thu, Mar 31, 2011 at 12:46 PM, Adam GROSZER wrote: > After investigating FileStorage a bit, I found that GC runs on objects, > but pack later by transactions. That means that if there's a bigger-ish > transaction, we can't get rid of it until all of it's objects are GCed > (or superseeded by newer states). > > Is that correct? I think so. > I think yes, so an idea might be to split up > transactions to one transaction per object state somehow and pack again. > This would definitely work only offline of course. I think it would be interesting to gather some statistics on this. How often does this actually happen, and how much "orphaned" data could one get rid of. Implementing something that iterates over transactions, takes the "live" objects of it and writes them into a new transaction should be possible without taking the database offline. > How is this handled by relstorage? RelStorage does the same. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packing ZODB
On Thu, Mar 31, 2011 at 6:46 AM, Adam GROSZER wrote: > Hello, > > After investigating FileStorage a bit, I found that GC runs on objects, > but pack later by transactions. That means that if there's a bigger-ish > transaction, we can't get rid of it until all of it's objects are GCed > (or superseeded by newer states). The transaction stays around, but non-current records in the transaction and records for garbage objects are removes. Who cares if (some trace of) a transaction stays around? > Is that correct? The transaction itself lingers only as long as there are records remaining, but not all of the records stay around. I'm not sure what you meant above, but if you think all of the records in the transaction stay around, the answers is no. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] packing ZODB
Hello, After investigating FileStorage a bit, I found that GC runs on objects, but pack later by transactions. That means that if there's a bigger-ish transaction, we can't get rid of it until all of it's objects are GCed (or superseeded by newer states). Is that correct? I think yes, so an idea might be to split up transactions to one transaction per object state somehow and pack again. This would definitely work only offline of course. How is this handled by relstorage? -- Best regards, Adam GROSZER -- Quote of the day: The most beautiful thing we can experience is the mysterious. It is the source of all true art and science. - Albert Einstein ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
