On Thu, Aug 22, 2013 at 03:40:41PM -0600, Sean Upton wrote: > Caveat to this below. > > On Mon, Jul 15, 2013 at 7:54 AM, Marius Gedminas <mar...@gedmin.as> wrote: > > > On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote: > > > We need to move a considerable number of persistent objects (~40K) from > > > one module to another, and we were wondering what is the best strategy > > > to follow. > > ... > > > Has anyone ever done anything like this? Which approach have you > > > followed? Any suggestions? > > > > 1. Make sure all the classes are still importable from the old location > > ('from newmodule import MyPersistentSomething # BBB') > > > > That is sufficient to make it work. If you also want to eradicate all > > references to the old module name from your ZODB (e.g. because you'd > > like to remove the BBB import), then proceed to step 2: > > > > 2. Write a script that loads every instance of this class and does a > > > > obj._p_activate() # actually not sure this is required, but won't hurt > > obj._p_changed = True > > > > and then commit the transaction. Do the commit multiple times, after > > each batch of several hundred objects, to avoid excessive memory usage. > > Also be sure to handle conflict errors and retry that batch if you're > > running this script on the live system. Finding all instances is left > > as an exercise for the reader (sometimes findObjectsProviding() helps, > > if you use nested containers everywhere; sometimes application-specific > > logic works best; sometimes you end up having to use ZODB iterators to > > loop through every single object in the DB -- I believe zodbupdate does > > that.) > > > > This works only partially, AFAICT. It will update the stored class name of > a persistent object. It will not seek out and change the class name of the > object in a reference. Your step 3 (removing the BBB code) can break > things (broken objects) as a result of this. Fixing persistent objects is > half the battle if you get broken references to them.
Ouch, you're absolutely right. > You can verify that this is incomplete by creating a PersistentMapping of > some simple objects, perform the trick above, commit, then pack. You will > still have the old BBB classname stored in the database in the references, > unless you do a _p_changed=1 on the mapping containing/referencing the > items as well (then commit). > > I am not sure how to walk/iterate all oids for all transactions to get all > possible referencing objects (I assume this is storage-specific, maybe > building a reference map like Lawrence Rowe has done ). This might be > necessary to update the referencing objects? ZODB has an iterator API that allows you to access all the objects. I think SchoolTool used to have an evolution script that did precisely this (walk all the objects, set _p_changed to force a write) to solve this problem. And then there's https://pypi.python.org/pypi/zodbupdate which sounds exactly like the tool you want for this. >  http://plone.org/documentation/kb/debug-zodb-bloat/inspectZodbUtils.py Marius Gedminas -- Killing gnome-session is likely to free substantial amounts of memory, but the user's gratitude may be surprisingly limited. -- Jonathan Corbet in a LWN article about the OOM killer
Description: Digital signature
_______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev