Re: [ZODB-Dev] Changing namespace - best strategy
On 08/23/2013 09:07 AM, Marius Gedminas wrote: And then there's https://pypi.python.org/pypi/zodbupdate which sounds exactly like the tool you want for this. Yeah, that definitely works. You just got to get the moving classes/definitions right. -- Best regards, Adam GROSZER -- Quote of the day: Knowledge by suffering entereth, And life is perfected by death. - Elizabeth Barrett Browning ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Thu, Aug 22, 2013 at 03:40:41PM -0600, Sean Upton wrote: > Caveat to this below. > > On Mon, Jul 15, 2013 at 7:54 AM, Marius Gedminas wrote: > > > On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote: > > > We need to move a considerable number of persistent objects (~40K) from > > > one module to another, and we were wondering what is the best strategy > > > to follow. > > ... > > > Has anyone ever done anything like this? Which approach have you > > > followed? Any suggestions? > > > > 1. Make sure all the classes are still importable from the old location > >('from newmodule import MyPersistentSomething # BBB') > > > > That is sufficient to make it work. If you also want to eradicate all > > references to the old module name from your ZODB (e.g. because you'd > > like to remove the BBB import), then proceed to step 2: > > > > 2. Write a script that loads every instance of this class and does a > > > >obj._p_activate() # actually not sure this is required, but won't hurt > >obj._p_changed = True > > > > and then commit the transaction. Do the commit multiple times, after > > each batch of several hundred objects, to avoid excessive memory usage. > > Also be sure to handle conflict errors and retry that batch if you're > > running this script on the live system. Finding all instances is left > > as an exercise for the reader (sometimes findObjectsProviding() helps, > > if you use nested containers everywhere; sometimes application-specific > > logic works best; sometimes you end up having to use ZODB iterators to > > loop through every single object in the DB -- I believe zodbupdate does > > that.) > > > > This works only partially, AFAICT. It will update the stored class name of > a persistent object. It will not seek out and change the class name of the > object in a reference. Your step 3 (removing the BBB code) can break > things (broken objects) as a result of this. Fixing persistent objects is > half the battle if you get broken references to them. Ouch, you're absolutely right. > You can verify that this is incomplete by creating a PersistentMapping of > some simple objects, perform the trick above, commit, then pack. You will > still have the old BBB classname stored in the database in the references, > unless you do a _p_changed=1 on the mapping containing/referencing the > items as well (then commit). > > I am not sure how to walk/iterate all oids for all transactions to get all > possible referencing objects (I assume this is storage-specific, maybe > building a reference map like Lawrence Rowe has done [1]). This might be > necessary to update the referencing objects? ZODB has an iterator API that allows you to access all the objects. I think SchoolTool used to have an evolution script that did precisely this (walk all the objects, set _p_changed to force a write) to solve this problem. And then there's https://pypi.python.org/pypi/zodbupdate which sounds exactly like the tool you want for this. > [1] http://plone.org/documentation/kb/debug-zodb-bloat/inspectZodbUtils.py Marius Gedminas -- Killing gnome-session is likely to free substantial amounts of memory, but the user's gratitude may be surprisingly limited. -- Jonathan Corbet in a LWN article about the OOM killer signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
Caveat to this below. On Mon, Jul 15, 2013 at 7:54 AM, Marius Gedminas wrote: > On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote: > > We need to move a considerable number of persistent objects (~40K) from > > one module to another, and we were wondering what is the best strategy > > to follow. > ... > > Has anyone ever done anything like this? Which approach have you > > followed? Any suggestions? > > 1. Make sure all the classes are still importable from the old location >('from newmodule import MyPersistentSomething # BBB') > > That is sufficient to make it work. If you also want to eradicate all > references to the old module name from your ZODB (e.g. because you'd > like to remove the BBB import), then proceed to step 2: > > 2. Write a script that loads every instance of this class and does a > >obj._p_activate() # actually not sure this is required, but won't hurt >obj._p_changed = True > > and then commit the transaction. Do the commit multiple times, after > each batch of several hundred objects, to avoid excessive memory usage. > Also be sure to handle conflict errors and retry that batch if you're > running this script on the live system. Finding all instances is left > as an exercise for the reader (sometimes findObjectsProviding() helps, > if you use nested containers everywhere; sometimes application-specific > logic works best; sometimes you end up having to use ZODB iterators to > loop through every single object in the DB -- I believe zodbupdate does > that.) > This works only partially, AFAICT. It will update the stored class name of a persistent object. It will not seek out and change the class name of the object in a reference. Your step 3 (removing the BBB code) can break things (broken objects) as a result of this. Fixing persistent objects is half the battle if you get broken references to them. You can verify that this is incomplete by creating a PersistentMapping of some simple objects, perform the trick above, commit, then pack. You will still have the old BBB classname stored in the database in the references, unless you do a _p_changed=1 on the mapping containing/referencing the items as well (then commit). I am not sure how to walk/iterate all oids for all transactions to get all possible referencing objects (I assume this is storage-specific, maybe building a reference map like Lawrence Rowe has done [1]). This might be necessary to update the referencing objects? [1] http://plone.org/documentation/kb/debug-zodb-bloat/inspectZodbUtils.py Sean ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Tue, Jul 16, 2013 at 08:54:55PM +0200, Pedro Ferreira wrote: > Hello, > > Thanks for your answer. > > > 1. Make sure all the classes are still importable from the old location > >('from newmodule import MyPersistentSomething # BBB') > > So, I assume this is based on the classFactory hook/dictionary solution? > Or am I misunderstanding it? No, this is based on ZODB's use of pickles. The database stores the fully-qualified name of the module and class of each object, so if you make the classes importable using the old module and class name, the objects can still be loaded. Marius Gedminas -- Moore's Law, I need hardly remind a top-notch industry professional like you, states that as the density of silicon circuitry doubles, the probability of you not being able to find some sensibly-priced extra memory to fit your old lappy approaches 1.0. -- Verity Stob signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
Hello, Thanks for your answer. > 1. Make sure all the classes are still importable from the old location >('from newmodule import MyPersistentSomething # BBB') So, I assume this is based on the classFactory hook/dictionary solution? Or am I misunderstanding it? Cheers, Pedro -- Pedro Ferreira http://pferreir.github.com ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote: > We need to move a considerable number of persistent objects (~40K) from > one module to another, and we were wondering what is the best strategy > to follow. ... > Has anyone ever done anything like this? Which approach have you > followed? Any suggestions? 1. Make sure all the classes are still importable from the old location ('from newmodule import MyPersistentSomething # BBB') That is sufficient to make it work. If you also want to eradicate all references to the old module name from your ZODB (e.g. because you'd like to remove the BBB import), then proceed to step 2: 2. Write a script that loads every instance of this class and does a obj._p_activate() # actually not sure this is required, but won't hurt obj._p_changed = True and then commit the transaction. Do the commit multiple times, after each batch of several hundred objects, to avoid excessive memory usage. Also be sure to handle conflict errors and retry that batch if you're running this script on the live system. Finding all instances is left as an exercise for the reader (sometimes findObjectsProviding() helps, if you use nested containers everywhere; sometimes application-specific logic works best; sometimes you end up having to use ZODB iterators to loop through every single object in the DB -- I believe zodbupdate does that.) 3. (Optional) Remove the BBB import added in step 1. HTH, Marius Gedminas -- Voodoo Programming: Things programmers do that they know shouldn't work but they try anyway, and which sometimes actually work, such as recompiling everything. -- Karl Lehenbauer signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev