Re: [ZODB-Dev] notes re trying to run ZODB on PyPy
> Most were temporary hacks and the rest have no tests yet, so not checked > in anywhere. But here's (attached) the diff from my working directory. Thanks a lot for posting this, it seems like a good first step in abstracting ZODB from CPython. This reminds me of this particular presentation: http://talk.quintagroup.com/blogs/myroslav/zope-at-gae Does anyone know if any code has been made available? Best regards, Pedro -- José Pedro Ferreira Software Developer, Indico Project http://indico-software.org +---+ + '``'--- `+ CERN - European Organization for Nuclear Research + |CERN| / + 1211 Geneve 23, Switzerland + ..__. \. + IT-UDS-AVC + \\___.\ + Office: 513-1-005 + /+ Tel. +41227677159 +---+ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] notes re trying to run ZODB on PyPy
On 9/25/11 8:17 AM, Tres Seaver wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 09/24/2011 08:02 PM, David Glick wrote:
Alan Runyan asked me to post my notes from my attempt to get ZODB
running on PyPy. It's very much an experimental work in progress
(that I got distracted from), but hopefully this is at least useful
to anyone else who wants to attempt the same thing.
I first tried building ZODB trunk (with C extensions). Ran into the
following issues: * PyPy is missing an implementation of the
_Py_ForgetReference macro which is used by ZODB's
persistent/cPickleCache.c * The check_argument_cmp check in
BTrees/objectkeymacros.h fails for the root object of the database on
PyPy * I got a PyPy exception re wrap_objobjargspec which appeared to
be some issue with using the __setitem__ slot of an extension type
At this point I gave up on trying to build the C extensions, and
instead did the following: 1. Started with Jim's python-btrees branch
(http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) 2.
Replaced the 'persistent' package with a checkout of Tres'
pure-Python persistent implementation
(http://svn.zope.org/repos/main/persistent/trunk/persistent) 3. Set
exts = [] in setup.py to disable all C extensions when installing.
Next I worked around some issues in the pure-Python branches: *
BTrees.___BTree.fsBucket class was missing toString and fromString
methods, which I implemented. * persistent.picklecache.PickleCache
class was missing update_object_size_estimation method (and indeed
the whole limit-cache-size-by-bytes feature). I added it as a no-op
stub. * persistent.picklecache.PickleCache.__setitem__ was raising a
KeyError for a duplicate oid even when trying to set the same object
already stored under that key * persistent.TimeStamp did not give the
Python implementation, which I fixed by an import of
persistent.timestamp + module alias if persistent.TimeStamp (the C
module) is missing * persistent.pyPersistence.Persistent's __new__
needs to accept *args and **kw
At this point I was able to start up the Pyramid ZODB scaffold and
add an object to the DB root.
Next I tried to get zodbshootout running as a benchmark. I didn't get
this working, as I got stuck on an issue with the pure-Python
PersistentMapping getting committed without its object state
(appeared to be something to do with failing to get unghosted before
it gets committed). Along the way, I noticed a couple
incompatibilities with PyPy because PyPy implements the cPickle
module as an alias to the pure-Python pickle module. This leads to
some problems due to slight semantic differences between cPickle and
pickle. In particular, cPickle.Pickler accepts a lone protocol
argument; pickle.Pickler does not (leading to an error in
ZEO.zrpc.marshal.encode). And pickle.Unpickler does not have a noload
method, which ZODB.serialize.referencesf uses to obtain the
referenced oids from a pickle without the overhead of unpickling.
That's as far as I got. Hope this is helpful to the next guy. :)
Cool, great work. Are the changes you needed to make available as
patchers, or checked in?
Most were temporary hacks and the rest have no tests yet, so not checked
in anywhere. But here's (attached) the diff from my working directory.
--
David Glick
Web Developer
davidgl...@groundwire.org
206.286.1235x32
Online tools and strategies for the environmental movement.
Sign up for our newsletter: http://www.groundwire.org/email-capture
Index: src/persistent/picklecache.py
===
--- src/persistent/picklecache.py (revision 122718)
+++ src/persistent/picklecache.py (working copy)
@@ -65,7 +65,8 @@
raise ValueError('OID must be string: %s' % oid)
# XXX
if oid in self.persistent_classes or oid in self.data:
-raise KeyError('Duplicate OID: %s' % oid)
+if self.data[oid] is not value:
+raise KeyError('Duplicate OID: %s' % oid)
if type(value) is type:
self.persistent_classes[oid] = value
else:
@@ -255,3 +256,5 @@
elif oid in self.persistent_classes:
del self.persistent_classes[oid]
+def update_object_size_estimation(self, oid, new_size):
+pass
Index: src/persistent/pyPersistence.py
===
--- src/persistent/pyPersistence.py (revision 122718)
+++ src/persistent/pyPersistence.py (working copy)
@@ -80,7 +80,7 @@
__slots__ = ('__jar', '__oid', '__serial', '__flags', '__size')
implements(IPersistent)
-def __new__(cls):
+def __new__(cls, *args, **kw):
inst = super(Persistent, cls).__new__(cls)
inst.__jar = None
inst.__oid = None
Index: src/persistent/__init__.py
===
--- src/persistent/__init__.py (revision 122718)
+++ src/persistent/__in
Re: [ZODB-Dev] notes re trying to run ZODB on PyPy
On Sat, Sep 24, 2011 at 8:02 PM, David Glick wrote: ... > At this point I gave up on trying to build the C extensions, and instead > did the following: > 1. Started with Jim's python-btrees branch > (http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) This is a great place for someone to help out. :) Just needs finishing IIRC. :) ... > At this point I was able to start up the Pyramid ZODB scaffold and add > an object to the DB root. ... > That's as far as I got. Hope this is helpful to the next guy. :) Very cool. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] notes re trying to run ZODB on PyPy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/24/2011 08:02 PM, David Glick wrote: > Alan Runyan asked me to post my notes from my attempt to get ZODB > running on PyPy. It's very much an experimental work in progress > (that I got distracted from), but hopefully this is at least useful > to anyone else who wants to attempt the same thing. > > I first tried building ZODB trunk (with C extensions). Ran into the > following issues: * PyPy is missing an implementation of the > _Py_ForgetReference macro which is used by ZODB's > persistent/cPickleCache.c * The check_argument_cmp check in > BTrees/objectkeymacros.h fails for the root object of the database on > PyPy * I got a PyPy exception re wrap_objobjargspec which appeared to > be some issue with using the __setitem__ slot of an extension type > > At this point I gave up on trying to build the C extensions, and > instead did the following: 1. Started with Jim's python-btrees branch > (http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) 2. > Replaced the 'persistent' package with a checkout of Tres' > pure-Python persistent implementation > (http://svn.zope.org/repos/main/persistent/trunk/persistent) 3. Set > exts = [] in setup.py to disable all C extensions when installing. > > Next I worked around some issues in the pure-Python branches: * > BTrees.___BTree.fsBucket class was missing toString and fromString > methods, which I implemented. * persistent.picklecache.PickleCache > class was missing update_object_size_estimation method (and indeed > the whole limit-cache-size-by-bytes feature). I added it as a no-op > stub. * persistent.picklecache.PickleCache.__setitem__ was raising a > KeyError for a duplicate oid even when trying to set the same object > already stored under that key * persistent.TimeStamp did not give the > Python implementation, which I fixed by an import of > persistent.timestamp + module alias if persistent.TimeStamp (the C > module) is missing * persistent.pyPersistence.Persistent's __new__ > needs to accept *args and **kw > > At this point I was able to start up the Pyramid ZODB scaffold and > add an object to the DB root. > > Next I tried to get zodbshootout running as a benchmark. I didn't get > this working, as I got stuck on an issue with the pure-Python > PersistentMapping getting committed without its object state > (appeared to be something to do with failing to get unghosted before > it gets committed). Along the way, I noticed a couple > incompatibilities with PyPy because PyPy implements the cPickle > module as an alias to the pure-Python pickle module. This leads to > some problems due to slight semantic differences between cPickle and > pickle. In particular, cPickle.Pickler accepts a lone protocol > argument; pickle.Pickler does not (leading to an error in > ZEO.zrpc.marshal.encode). And pickle.Unpickler does not have a noload > method, which ZODB.serialize.referencesf uses to obtain the > referenced oids from a pickle without the overhead of unpickling. > > That's as far as I got. Hope this is helpful to the next guy. :) Cool, great work. Are the changes you needed to make available as patchers, or checked in? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5/RgAACgkQ+gerLs4ltQ7NmgCg2n4GT1wQWxPaRNWCguOtAb8q OQ4AoKcbDGep0ig5ZL/x+/m4hNeAVyuq =KtIZ -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
