Re: [ZODB-Dev] Changing namespace - best strategy
On Thu, Aug 22, 2013 at 03:40:41PM -0600, Sean Upton wrote:
Caveat to this below.
On Mon, Jul 15, 2013 at 7:54 AM, Marius Gedminas mar...@gedmin.as wrote:
On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote:
We need to move a considerable number of persistent objects (~40K) from
one module to another, and we were wondering what is the best strategy
to follow.
...
Has anyone ever done anything like this? Which approach have you
followed? Any suggestions?
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
That is sufficient to make it work. If you also want to eradicate all
references to the old module name from your ZODB (e.g. because you'd
like to remove the BBB import), then proceed to step 2:
2. Write a script that loads every instance of this class and does a
obj._p_activate() # actually not sure this is required, but won't hurt
obj._p_changed = True
and then commit the transaction. Do the commit multiple times, after
each batch of several hundred objects, to avoid excessive memory usage.
Also be sure to handle conflict errors and retry that batch if you're
running this script on the live system. Finding all instances is left
as an exercise for the reader (sometimes findObjectsProviding() helps,
if you use nested containers everywhere; sometimes application-specific
logic works best; sometimes you end up having to use ZODB iterators to
loop through every single object in the DB -- I believe zodbupdate does
that.)
This works only partially, AFAICT. It will update the stored class name of
a persistent object. It will not seek out and change the class name of the
object in a reference. Your step 3 (removing the BBB code) can break
things (broken objects) as a result of this. Fixing persistent objects is
half the battle if you get broken references to them.
Ouch, you're absolutely right.
You can verify that this is incomplete by creating a PersistentMapping of
some simple objects, perform the trick above, commit, then pack. You will
still have the old BBB classname stored in the database in the references,
unless you do a _p_changed=1 on the mapping containing/referencing the
items as well (then commit).
I am not sure how to walk/iterate all oids for all transactions to get all
possible referencing objects (I assume this is storage-specific, maybe
building a reference map like Lawrence Rowe has done [1]). This might be
necessary to update the referencing objects?
ZODB has an iterator API that allows you to access all the objects.
I think SchoolTool used to have an evolution script that did precisely
this (walk all the objects, set _p_changed to force a write) to solve
this problem.
And then there's https://pypi.python.org/pypi/zodbupdate which sounds
exactly like the tool you want for this.
[1] http://plone.org/documentation/kb/debug-zodb-bloat/inspectZodbUtils.py
Marius Gedminas
--
Killing gnome-session is likely to free substantial amounts of memory, but the
user's gratitude may be surprisingly limited.
-- Jonathan Corbet in a LWN article about the OOM killer
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On 08/23/2013 09:07 AM, Marius Gedminas wrote: And then there's https://pypi.python.org/pypi/zodbupdate which sounds exactly like the tool you want for this. Yeah, that definitely works. You just got to get the moving classes/definitions right. -- Best regards, Adam GROSZER -- Quote of the day: Knowledge by suffering entereth, And life is perfected by death. - Elizabeth Barrett Browning ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
Caveat to this below.
On Mon, Jul 15, 2013 at 7:54 AM, Marius Gedminas mar...@gedmin.as wrote:
On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote:
We need to move a considerable number of persistent objects (~40K) from
one module to another, and we were wondering what is the best strategy
to follow.
...
Has anyone ever done anything like this? Which approach have you
followed? Any suggestions?
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
That is sufficient to make it work. If you also want to eradicate all
references to the old module name from your ZODB (e.g. because you'd
like to remove the BBB import), then proceed to step 2:
2. Write a script that loads every instance of this class and does a
obj._p_activate() # actually not sure this is required, but won't hurt
obj._p_changed = True
and then commit the transaction. Do the commit multiple times, after
each batch of several hundred objects, to avoid excessive memory usage.
Also be sure to handle conflict errors and retry that batch if you're
running this script on the live system. Finding all instances is left
as an exercise for the reader (sometimes findObjectsProviding() helps,
if you use nested containers everywhere; sometimes application-specific
logic works best; sometimes you end up having to use ZODB iterators to
loop through every single object in the DB -- I believe zodbupdate does
that.)
This works only partially, AFAICT. It will update the stored class name of
a persistent object. It will not seek out and change the class name of the
object in a reference. Your step 3 (removing the BBB code) can break
things (broken objects) as a result of this. Fixing persistent objects is
half the battle if you get broken references to them.
You can verify that this is incomplete by creating a PersistentMapping of
some simple objects, perform the trick above, commit, then pack. You will
still have the old BBB classname stored in the database in the references,
unless you do a _p_changed=1 on the mapping containing/referencing the
items as well (then commit).
I am not sure how to walk/iterate all oids for all transactions to get all
possible referencing objects (I assume this is storage-specific, maybe
building a reference map like Lawrence Rowe has done [1]). This might be
necessary to update the referencing objects?
[1] http://plone.org/documentation/kb/debug-zodb-bloat/inspectZodbUtils.py
Sean
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Tue, Jul 16, 2013 at 08:54:55PM +0200, Pedro Ferreira wrote:
Hello,
Thanks for your answer.
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
So, I assume this is based on the classFactory hook/dictionary solution?
Or am I misunderstanding it?
No, this is based on ZODB's use of pickles. The database stores the
fully-qualified name of the module and class of each object, so if you
make the classes importable using the old module and class name, the
objects can still be loaded.
Marius Gedminas
--
Moore's Law, I need hardly remind a top-notch industry professional like you,
states that as the density of silicon circuitry doubles, the probability of you
not being able to find some sensibly-priced extra memory to fit your old lappy
approaches 1.0.
-- Verity Stob
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
Hello,
Thanks for your answer.
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
So, I assume this is based on the classFactory hook/dictionary solution?
Or am I misunderstanding it?
Cheers,
Pedro
--
Pedro Ferreira
http://pferreir.github.com
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Changing namespace - best strategy
Hello everyone, We need to move a considerable number of persistent objects (~40K) from one module to another, and we were wondering what is the best strategy to follow. Our database contains around 90M objects, which is a pain to run `zodbupdate` on. So far these are the options that we thought of: * Running `zodbupdate`, possibly restricting the target object set to the places where we know that references to the class in question exist (otherwise it takes REALLY long); * Using the `classFactory`/`find_global` hook and basically converting the objects/refs to the new class as they are requested; The first one seems cleaner and less prone to problems in the future, but it will take ages (our current benchmarks point towards 1 day, although we could try to reduce that by limiting the search scope). The second one seems much easier to execute, but the fact that objects/references to the old and new classes will co-exist kind of bothers me. Has anyone ever done anything like this? Which approach have you followed? Any suggestions? Thanks a lot in advance, Cheers, Pedro P.S.: I am sending this from my private mail because this list seems to have been blacklisted in some servers, including my organization's. -- Pedro Ferreira http://pferreir.github.com ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote:
We need to move a considerable number of persistent objects (~40K) from
one module to another, and we were wondering what is the best strategy
to follow.
...
Has anyone ever done anything like this? Which approach have you
followed? Any suggestions?
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
That is sufficient to make it work. If you also want to eradicate all
references to the old module name from your ZODB (e.g. because you'd
like to remove the BBB import), then proceed to step 2:
2. Write a script that loads every instance of this class and does a
obj._p_activate() # actually not sure this is required, but won't hurt
obj._p_changed = True
and then commit the transaction. Do the commit multiple times, after
each batch of several hundred objects, to avoid excessive memory usage.
Also be sure to handle conflict errors and retry that batch if you're
running this script on the live system. Finding all instances is left
as an exercise for the reader (sometimes findObjectsProviding() helps,
if you use nested containers everywhere; sometimes application-specific
logic works best; sometimes you end up having to use ZODB iterators to
loop through every single object in the DB -- I believe zodbupdate does
that.)
3. (Optional) Remove the BBB import added in step 1.
HTH,
Marius Gedminas
--
Voodoo Programming: Things programmers do that they know shouldn't work but
they try anyway, and which sometimes actually work, such as recompiling
everything.
-- Karl Lehenbauer
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
