Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
Tim Peters wrote: [Tim Peters] Stephan Richter was wrestling with a migration script a couple months ago. I don't know whether he packaged the code he ended up with for reuse. Here's a record of the IRC chat in which details got worked out: http://zope3.pov.lt/irclogs/%23zope3-dev.2005-08-25.log.html [Stephan Richter] The resulting code from the discussion is publically available in the SchoolTool source: http://source.schooltool.org/viewcvs/trunk/schooltool/src/schooltool/genera tions/evolve5.py?rev=4984view=auto [Syver Enstad] Thank you Stefan. I am right if I suppose that the reason this fixes all references is that it saves all persistent objects in the database? Basically, but read the IRC log for details: it (re)saves the current revision of all objects not in a version. It does not change non-current revisions of objects, nor any revisions of objects in a version. This is because the FileStorage iterator only knows about current non-version revisions. I don't use versions so that's fine. The IRC log was very illuminating BTW. It seemed to basically be going through exactly the same issues that I have been going through. Will this also fix references to persistent objects in non persistent objects stored in the database? If (and only if) they're reachable from the root object following a chain of non-version current revisions. You can't store a non-persistent object N directly in ZODB -- the only way N can get in is by being attached to a persistent object P. Then N's entire state is included in the pickle for P. So when P is (re)saved, the entire state of N is recomputed, including (of course) all references to persistent objects (if any) contained in N. I see, that sounds good to me. For now I am basically one happy guy. Thanks to everyone for responding and helping me out with my problem migrating. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
Tim Peters wrote: [David Binger] Is there something to prevent your update script from setting _p_changed = 1 on every persistent object that contains a reference to an instance of the moved class? I think that if you do that and commit and pack the database, the old module/class references will be purged. If you do this, the sys.modules hacking only has to happen in the update script. Stephan Richter was wrestling with a migration script a couple months ago. I don't know whether he packaged the code he ended up with for reuse. Here's a record of the IRC chat in which details got worked out: http://zope3.pov.lt/irclogs/%23zope3-dev.2005-08-25.log.html Mostly because of that, ZODB 3.6 changed so that if you set _p_changed to a true value on a ghost object, the state for the ghost is loaded. Before 3.6, setting _p_changed to a true value on a ghost object is ignored (no exception, and the object remains a ghost). In short, setting _p_changed may not be enough. Before 3.6, _p_activate() should be called too (before setting _p_changed; see the IRC log for details). BTW, no version of ZODB 3.6 has been released publicly yet, although current Zope 2.9 and 3.2 development use an internal 3.6.0b3 release (i.e., it's solid enough, but I haven't had time to make/test/publish tarballs/installers/news/etc). [Syver Enstad] ... What I don't like is that this will break for the following case as far as I see. firstModule.First renamed - secondModule.Second Some time later in the database life: New class: firstModule.First. Will be hidden by the machinery that hacks in secondModule.Second under the old name firstModule.First. You'll see I had the same objection in the log referenced above. A one-shot migration script seems doable, but I don't think there's a wholly sane approach on the horizon for doing live, lazy, incremental migration slobbering over decades ;-) That's quite okay. I am only searching for a way to perform atomic offline upgrades with an upgrade script, so that my source code can stay clean not having to care about what the database looked like N versions ago. Except for my upgrade machinery of course. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
Stephan Richter wrote: On Wednesday 09 November 2005 11:00, Tim Peters wrote: Stephan Richter was wrestling with a migration script a couple months ago. I don't know whether he packaged the code he ended up with for reuse. Here's a record of the IRC chat in which details got worked out: http://zope3.pov.lt/irclogs/%23zope3-dev.2005-08-25.log.html The resulting code from the discussion is publically available in the SchoolTool source: http://source.schooltool.org/viewcvs/trunk/schooltool/src/schooltool/generations/evolve5.py?rev=4984view=auto Regards, Stephan Thank you Stefan. I am right if I suppose that the reason this fixes all references is that it saves all persistent objects in the database? Will this also fix references to persistent objects in non persistent objects stored in the database? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
Lennart Regebro wrote: This may not help, but anyway. :) In Zope, I did this: I moved the class to the module I wanted it. I also kept a dummy class in the old place like so: from newplace import class so that the old objects worked. Then I wrote a script to go through the ZODB and recreate every object. I then could remove the dummy import. That sounds good. How do you recreate every object in the ZODB? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On Nov 10, 2005, at 5:09 AM, Syver Enstad wrote: Thank you Stefan. I am right if I suppose that the reason this fixes all references is that it saves all persistent objects in the database? Will this also fix references to persistent objects in non persistent objects stored in the database? Yes. All such references appear somewhere in the pickle of a persistent object. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
Stephan Richter wrote: On Wednesday 09 November 2005 11:00, Tim Peters wrote: Stephan Richter was wrestling with a migration script a couple months ago. I don't know whether he packaged the code he ended up with for reuse. Here's a record of the IRC chat in which details got worked out: http://zope3.pov.lt/irclogs/%23zope3-dev.2005-08-25.log.html The resulting code from the discussion is publically available in the SchoolTool source: http://source.schooltool.org/viewcvs/trunk/schooltool/src/schooltool/generations/evolve5.py?rev=4984view=auto Regards, Stephan Yuhuu!! It seems to work. So the key is to force activation and repickling of every persistent object in the database, then all the modulename, classname references are updated to the current. It certainly seems that I can use my original approach of overriding classFactory in DB for the duration of the upgrade script. Very nice. I almost did the same thing as the evolve function but I only changed the objects who where of the class that I was moving to another module. That didn't work in 3.5.1 but worked in 3.2. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Migrating classes in ZODB 3.5.1
[Tim Peters] Stephan Richter was wrestling with a migration script a couple months ago. I don't know whether he packaged the code he ended up with for reuse. Here's a record of the IRC chat in which details got worked out: http://zope3.pov.lt/irclogs/%23zope3-dev.2005-08-25.log.html [Stephan Richter] The resulting code from the discussion is publically available in the SchoolTool source: http://source.schooltool.org/viewcvs/trunk/schooltool/src/schooltool/genera tions/evolve5.py?rev=4984view=auto [Syver Enstad] Thank you Stefan. I am right if I suppose that the reason this fixes all references is that it saves all persistent objects in the database? Basically, but read the IRC log for details: it (re)saves the current revision of all objects not in a version. It does not change non-current revisions of objects, nor any revisions of objects in a version. This is because the FileStorage iterator only knows about current non-version revisions. Will this also fix references to persistent objects in non persistent objects stored in the database? If (and only if) they're reachable from the root object following a chain of non-version current revisions. You can't store a non-persistent object N directly in ZODB -- the only way N can get in is by being attached to a persistent object P. Then N's entire state is included in the pickle for P. So when P is (re)saved, the entire state of N is recomputed, including (of course) all references to persistent objects (if any) contained in N. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On 11/9/05, David Binger [EMAIL PROTECTED] wrote: On Nov 9, 2005, at 9:21 AM, Syver Enstad wrote: What I want to do is just update the persistent instance so that it will load from the new module/ new class after I have done a one time update of the database. I fear this will not work. For efficiency reasons, the class designator (usually a module, classname tuple) is stored both in the object itself and in persistent references. While you will be able to modify the object's class the persistent references will only change when the containers are stored into the ZODB. Is there something to prevent your update script from setting _p_changed = 1 on every persistent object that contains a reference to an instance of the moved class? I think that if you do that and commit and pack the database, the old module/class references will be purged. If you do this, the sys.modules hacking only has to happen in the update script. How do you find all the references? It seems like you would need to execute a transaction that iterated over every object in the storage and searched the pickle for references to the class. I can imagine the update transaction taking a really long time to run, which increases the possibility that it will get a conflict error. Maybe it's a resolvable conflict, but not using the current mechanism. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On 11/9/05, David Binger [EMAIL PROTECTED] wrote: On Nov 9, 2005, at 10:37 AM, Jeremy Hylton wrote: How do you find all the references? It seems like you would need to execute a transaction that iterated over every object in the storage and searched the pickle for references to the class. You do need to do this to find the references (using referencesf()?), but it doesn't necessarily need to happen in one giant transaction. The script just needs to (load and) re-pickle every instance with a reference while the sys.modules hack is in place. Commit each referring instance in a new transaction if you like, and conflict errors should not be a problem. I suppose you could run a separate thread that performed this conversion in parallel with other activities. There's a bit of a coordination problem to make sure you're done. It seems other application code would need to use the old class definition until the update was completed, but that means new references can be created while the update is running. So you'd need some sort of coordination to atomically switch from old class to new class when the update was completed. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
Dieter Maurer wrote: Syver Enstad wrote at 2005-11-7 11:51 +0100: In ZODB 3.2 I can replace the _classFactory method on DB and locate the renamed/moved class by other means (a lookup table that maps old module classname pairs to another class in another module). If I change attributes of the persistent instance it will be saved with the new class name instead of the old. I can't get this strategy to work under 3.5.1. It seems to work okay as long as I use the custom classFactory method (not _classFactory as in 3.2) but 3.5.1 doesn't seem to update the modulename, classname when I save the instance. What I want to do is just update the persistent instance so that it will load from the new module/ new class after I have done a one time update of the database. I fear this will not work. For efficiency reasons, the class designator (usually a module, classname tuple) is stored both in the object itself and in persistent references. While you will be able to modify the object's class the persistent references will only change when the containers are stored into the ZODB. Okay, so as far as I can see one has to either install a custom classfactory method on the DB instance that maps from old to new locations or hack sys.modules to refer to the new location by the old name. What I don't like is that this will break for the following case as far as I see. firstModule.First renamed - secondModule.Second Some time later in the database life: New class: firstModule.First. Will be hidden by the machinery that hacks in secondModule.Second under the old name firstModule.First. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Migrating classes in ZODB 3.5.1
[David Binger] Is there something to prevent your update script from setting _p_changed = 1 on every persistent object that contains a reference to an instance of the moved class? I think that if you do that and commit and pack the database, the old module/class references will be purged. If you do this, the sys.modules hacking only has to happen in the update script. Stephan Richter was wrestling with a migration script a couple months ago. I don't know whether he packaged the code he ended up with for reuse. Here's a record of the IRC chat in which details got worked out: http://zope3.pov.lt/irclogs/%23zope3-dev.2005-08-25.log.html Mostly because of that, ZODB 3.6 changed so that if you set _p_changed to a true value on a ghost object, the state for the ghost is loaded. Before 3.6, setting _p_changed to a true value on a ghost object is ignored (no exception, and the object remains a ghost). In short, setting _p_changed may not be enough. Before 3.6, _p_activate() should be called too (before setting _p_changed; see the IRC log for details). BTW, no version of ZODB 3.6 has been released publicly yet, although current Zope 2.9 and 3.2 development use an internal 3.6.0b3 release (i.e., it's solid enough, but I haven't had time to make/test/publish tarballs/installers/news/etc). [Syver Enstad] ... What I don't like is that this will break for the following case as far as I see. firstModule.First renamed - secondModule.Second Some time later in the database life: New class: firstModule.First. Will be hidden by the machinery that hacks in secondModule.Second under the old name firstModule.First. You'll see I had the same objection in the log referenced above. A one-shot migration script seems doable, but I don't think there's a wholly sane approach on the horizon for doing live, lazy, incremental migration slobbering over decades ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Migrating classes in ZODB 3.5.1
On Nov 9, 2005, at 10:37 AM, Jeremy Hylton wrote: How do you find all the references? It seems like you would need to execute a transaction that iterated over every object in the storage and searched the pickle for references to the class. You do need to do this to find the references (using referencesf()?), but it doesn't necessarily need to happen in one giant transaction. The script just needs to (load and) re-pickle every instance with a reference while the sys.modules hack is in place. Commit each referring instance in a new transaction if you like, and conflict errors should not be a problem. I can imagine the update transaction taking a really long time to run, which increases the possibility that it will get a conflict error. Maybe it's a resolvable conflict, but not using the current mechanism. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Migrating classes in ZODB 3.5.1
[Syver Enstad] In ZODB 3.2 I can replace the _classFactory method on DB and locate the renamed/moved class by other means (a lookup table that maps old module classname pairs to another class in another module). If I change attributes of the persistent instance it will be saved with the new class name instead of the old. I can't get this strategy to work under 3.5.1. It seems to work okay as long as I use the custom classFactory method (not _classFactory as in 3.2) but 3.5.1 doesn't seem to update the modulename, classname when I save the instance. What I want to do is just update the persistent instance so that it will load from the new module/ new class after I have done a one time update of the database. The ZODB.broken module has a rebuild function that might be the answer but I didn't understand how to use it to help me. If anybody on the list knows of any documentation regarding these issues please refer me to that. I'm not familiar with broken objects in newer ZODBs, and didn't find any docs apart from the docstrings in broken.py. Brief overviews of new features often appear in NEWS.txt, but the news for ZODB 3.3a3 just says New broken object support., so no luck there either. I don't have time now to try to figure it out, so I hope someone else can help. From the docstrings, it looks like the intent may be that you repair a broken object by poking old-new maps into sys.modules, effectively using that dict as your lookup table. But I haven't used them, and I'm not sure. Sorry! ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
