[ZODB-Dev] ExportImport.py: failing on import of extension class
In ExportImport._importDuringCommit() I found this little gem:
pfile = StringIO(data)
unpickler = Unpickler(pfile)
unpickler.persistent_load = persistent_load
newp = StringIO()
pickler = Pickler(newp, 1)
pickler.persistent_id = persistent_id
pickler.dump(unpickler.load())
pickler.dump(unpickler.load())
data = newp.getvalue()
What's with the two load-and-dump lines near the end?
I ask because I'm trying to get a .zexp imported, but I get this
exception during the *second* of those two lines:
Traceback (innermost last):
Module ZPublisher.Publish, line 101, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 39, in call_object
Module OFS.ObjectManager, line 543, in manage_importObject
Module OFS.ObjectManager, line 560, in _importObjectFromFile
Module ZODB.ExportImport, line 85, in importFile
Module ZODB.Transaction, line 241, in commit
Module ZODB.Transaction, line 356, in _commit_objects
Module ZODB.Connection, line 344, in commit
Module ZODB.ExportImport, line 153, in _importDuringCommit
Module copy_reg, line 92, in __newobj__
AttributeError: ('__new__', function __newobj__ at 0x2b9bb93cf140,
(extension class Products.Archetypes.Schema.Schema at
2b9bbdf6c9d0,))
If I switch from cPickle to Pickle, I get an extra two lines of
traceback:
Traceback (innermost last):
Module ZPublisher.Publish, line 101, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 39, in call_object
Module OFS.ObjectManager, line 543, in manage_importObject
Module OFS.ObjectManager, line 560, in _importObjectFromFile
Module ZODB.ExportImport, line 86, in importFile
Module ZODB.Transaction, line 241, in commit
Module ZODB.Transaction, line 356, in _commit_objects
Module ZODB.Connection, line 344, in commit
Module ZODB.ExportImport, line 153, in _importDuringCommit
Module pickle, line 872, in load
Module pickle, line 1153, in load_reduce
Module copy_reg, line 95, in __newobj__
AttributeError: __new__
Evidently, copy_reg.__newobj__() is for use with new-style classes.
But that's weird, because the guy that gave me this .zexp says that it
comes from a Zope 2.7 instance (Python 2.3, Plone 2.0) and I'm trying
to load it into an instance with the same versions (he gave me a
Products tarball too). And Zope 2.7 pre-dates the switch to new-style
extension classes...doesn't it??
So I don't understand what's happening. I'd expect the .zexp to
contain only old-style extension classes which did not have a __new__
method.
Would this be explained if I'm misinformed about the originating zope
version? I wish I had more access to the original host, but I
don't... I'm just trying to pick up the pieces with what I've been
given.
--
Paul Winkler
http://www.slinkp.com
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ExportImport.py: failing on import of extension class
On Apr 26, 2007, at 2:13 AM, Paul Winkler wrote: In ExportImport._importDuringCommit() I found this little gem: pfile = StringIO(data) unpickler = Unpickler(pfile) unpickler.persistent_load = persistent_load newp = StringIO() pickler = Pickler(newp, 1) pickler.persistent_id = persistent_id pickler.dump(unpickler.load()) pickler.dump(unpickler.load()) data = newp.getvalue() What's with the two load-and-dump lines near the end? It is transforming the pickles by assigning new object ids to the objects imported. The unpickler unpickles the pickles into an internal format that can be creates without actually creating the original objects. It collects object ids and reassigns them. The pickler then turns the internal data back into pickles with the new object ids. This is done in 2 steps because database records consist of 2 pickles. The first has enough information to create a ghost. The second pickle contains the object state. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ExportImport.py: failing on import of extension class
On Thu, Apr 26, 2007 at 10:10:17AM -0400, Jim Fulton wrote: On Apr 26, 2007, at 2:13 AM, Paul Winkler wrote: In ExportImport._importDuringCommit() I found this little gem: pfile = StringIO(data) unpickler = Unpickler(pfile) unpickler.persistent_load = persistent_load newp = StringIO() pickler = Pickler(newp, 1) pickler.persistent_id = persistent_id pickler.dump(unpickler.load()) pickler.dump(unpickler.load()) data = newp.getvalue() What's with the two load-and-dump lines near the end? It is transforming the pickles by assigning new object ids to the objects imported. The unpickler unpickles the pickles into an internal format that can be creates without actually creating the original objects. It collects object ids and reassigns them. The pickler then turns the internal data back into pickles with the new object ids. This is done in 2 steps because database records consist of 2 pickles. The first has enough information to create a ghost. The second pickle contains the object state. Ah, thanks! So that means my import is succeeding with the ghost and failing on object state. Any ideas how to troubleshoot the failure in copy_reg.__newobj__? Why is a pickle of an old-style extension class getting run through copy_reg.__newobj__() which only works on new-style classes? (Or am I misinterpreting the problem?) Is this likely a sign of software version skew? Or what? Traceback (innermost last): Module ZPublisher.Publish, line 101, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module OFS.ObjectManager, line 543, in manage_importObject Module OFS.ObjectManager, line 560, in _importObjectFromFile Module ZODB.ExportImport, line 86, in importFile Module ZODB.Transaction, line 241, in commit Module ZODB.Transaction, line 356, in _commit_objects Module ZODB.Connection, line 344, in commit Module ZODB.ExportImport, line 153, in _importDuringCommit Module pickle, line 872, in load Module pickle, line 1153, in load_reduce Module copy_reg, line 95, in __newobj__ AttributeError: __new__ -- Paul Winkler http://www.slinkp.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ExportImport.py: failing on import of extension class
On Thu, Apr 26, 2007 at 10:51:55AM -0400, Jim Fulton wrote: Old-style extension classes become new-style classes in Zope 2.8 and later. Extension Classes *are* new stype classes in Zope 2.8 and beyond. That's what I thought. But my .zexp came from Zope 2.7.8 instance, and I'm loading it in a Zope 2.7.9 instance. I see two possibilities: * the guy who told me it came from 2.7.8 was wrong. Does the evidence support this? * something else is wrong Are there any known explanations in this category? -- Paul Winkler http://www.slinkp.com ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ExportImport.py: failing on import of extension class
On Apr 26, 2007, at 11:17 AM, Paul Winkler wrote: On Thu, Apr 26, 2007 at 10:51:55AM -0400, Jim Fulton wrote: Old-style extension classes become new-style classes in Zope 2.8 and later. Extension Classes *are* new stype classes in Zope 2.8 and beyond. That's what I thought. But my .zexp came from Zope 2.7.8 instance, and I'm loading it in a Zope 2.7.9 instance. Ah. I see two possibilities: * the guy who told me it came from 2.7.8 was wrong. Does the evidence support this? * something else is wrong Are there any known explanations in this category? shrug / If you want to pursue this further, I suggest getting a hold of the database record (e.g. using our friend the debugger) and running it through one of the xml pickle variants to see if you get any additional clues about what's going on. Sorry I can't be more helpful, but I would have to spend time looking a the specific case and I can't spend that time. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ExportImport.py: failing on import of extension class
Paul Winkler wrote at 2007-4-26 02:13 -0400: In ExportImport._importDuringCommit() I found this little gem: pfile = StringIO(data) unpickler = Unpickler(pfile) unpickler.persistent_load = persistent_load newp = StringIO() pickler = Pickler(newp, 1) pickler.persistent_id = persistent_id pickler.dump(unpickler.load()) pickler.dump(unpickler.load()) data = newp.getvalue() What's with the two load-and-dump lines near the end? They effectively copy the data as a pickle to newp mapping the persistent ids appropriately. ... If I switch from cPickle to Pickle, I get an extra two lines of traceback: Traceback (innermost last): Module ZPublisher.Publish, line 101, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module OFS.ObjectManager, line 543, in manage_importObject Module OFS.ObjectManager, line 560, in _importObjectFromFile Module ZODB.ExportImport, line 86, in importFile Module ZODB.Transaction, line 241, in commit Module ZODB.Transaction, line 356, in _commit_objects Module ZODB.Connection, line 344, in commit Module ZODB.ExportImport, line 153, in _importDuringCommit Module pickle, line 872, in load Module pickle, line 1153, in load_reduce Module copy_reg, line 95, in __newobj__ AttributeError: __new__ Execute this in an interactive interpreter and check what cls in cls.__new__ is (using pdb.pm()). ... Evidently, copy_reg.__newobj__() is for use with new-style classes. But that's weird, because the guy that gave me this .zexp says that it comes from a Zope 2.7 instance (Python 2.3, Plone 2.0) and I'm trying to load it into an instance with the same versions (he gave me a Products tarball too). The python versions might differ. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Storage Interfaces
I've just checked in a fair bit of re-factoring to try to define reasonable storage interfaces. The results can be found at: http://svn.zope.org/ZODB/trunk/src/ZODB/interfaces.py?view=auto and http://svn.zope.org/ZODB/trunk/src/ZEO/interfaces.py?view=auto Some notes: - I added a new tpc_transaction method to allow ZEO to get some information it needs without groping for _transaction. Storage servers will still work with storages that don't have this method, but will log problems. Current storage implementations should add this method. - I'm not sure whether history, loadSerial, pack, and registerDB, should be in IStorage. I have a feeling that packing and other database-management tasks should be exposed some other way. history and loadSerial are fairly special, although they can be wildly useful. (IMO, there should be a history view for all Zope objects that lets one at least see the transaction history for an object even if looking at old data isn't supported.) Only very specialized storages need registerDB AFAICT. - See my earlier message about getExtensionMethods. - I was able to retire a poorly documented and understood method that was only needed by ZEO: loadEx. - Intentionally included version-support methods from the APIs. - I arranged that storages that don't support undo or versions don't need to provide any of the methods. I cleaned up various other sundry things along the way and identified some bugs, fixing at least one. Comments on the APIs are very welcome. Hopefully this will clarify some things. One of my text tasks will be to add some documentation about concurrency requirements. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
