[ZODB-Dev] Changing namespace - best strategy

[Pedro Ferreira]

Hello everyone,

We need to move a considerable number of persistent objects (~40K) from
one module to another, and we were wondering what is the best strategy
to follow.

Our database contains around 90M objects, which is a pain to run
`zodbupdate` on. So far these are the options that we thought of:

 * Running `zodbupdate`, possibly restricting the target object set to
   the places where we know that references to the class in question
   exist (otherwise it takes REALLY long);
 * Using the `classFactory`/`find_global` hook and basically converting
   the objects/refs to the new class as they are requested;

The first one seems cleaner and less prone to problems in the future,
but it will take ages (our current benchmarks point towards 1 day,
although we could try to reduce that by limiting the search scope). The
second one seems much easier to execute, but the fact that
objects/references to the old and new classes will co-exist kind of
bothers me.

Has anyone ever done anything like this? Which approach have you
followed? Any suggestions?

Thanks a lot in advance,

Cheers,


Pedro

P.S.: I am sending this from my private mail because this list seems to
have been blacklisted in some servers, including my organization's.

-- 

Pedro Ferreira
http://pferreir.github.com
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Changing namespace - best strategy

[Marius Gedminas]

On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote:
 We need to move a considerable number of persistent objects (~40K) from
 one module to another, and we were wondering what is the best strategy
 to follow.
...
 Has anyone ever done anything like this? Which approach have you
 followed? Any suggestions?

1. Make sure all the classes are still importable from the old location
   ('from newmodule import MyPersistentSomething # BBB')

That is sufficient to make it work.  If you also want to eradicate all
references to the old module name from your ZODB (e.g. because you'd
like to remove the BBB import), then proceed to step 2:

2. Write a script that loads every instance of this class and does a

   obj._p_activate()  # actually not sure this is required, but won't hurt
   obj._p_changed = True

and then commit the transaction.  Do the commit multiple times, after
each batch of several hundred objects, to avoid excessive memory usage.
Also be sure to handle conflict errors and retry that batch if you're
running this script on the live system.  Finding all instances is left
as an exercise for the reader (sometimes findObjectsProviding() helps,
if you use nested containers everywhere; sometimes application-specific
logic works best; sometimes you end up having to use ZODB iterators to
loop through every single object in the DB -- I believe zodbupdate does
that.)

3. (Optional) Remove the BBB import added in step 1.

HTH,
Marius Gedminas
-- 
Voodoo Programming:  Things programmers do that they know shouldn't work but
they try anyway, and which sometimes actually work, such as recompiling
everything.
-- Karl Lehenbauer


signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev