Re: [ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/04/2014 06:28 AM, Godefroid Chapelle wrote: > Le 03/02/14 20:53, Tres Seaver a écrit : >> I >> >> wish you hadn't pushed that -- some of these changes are definitely >> inappropriate on the 3.10 branch (adding an Acquisition dependency >> is definitely wrong). >> >> >> Tres. > > Acquisition is added as a test dependency. Any hint how to replicate > the bug without acquisition is welcome. Define a subclass of Persistent which emulates what Acquisition does, e.g.: from persistent import Persistent class Foo(Persistent): @property def _p_jar(self): # or whatever attribute trggers return object() > Which other change is inappropriate ? Adding MANIFEST.in on a release branch seems wrong to me (I don't like them anyway, and we *definitely* don't want to encourage instsall-from-a-github-generated-tarball on a release branch). Overall, these changes would have been better pushed on a separate branch, which could then be merged into the appropriate release branches *and* the trunk. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLxJukACgkQ+gerLs4ltQ4iEwCfSMrYBnSJbm9JzTJoqeb84lWk v2YAnRAqkPFJ5zcVscm8C4qymwSjeeQI =OEB0 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/03/2014 08:30 AM, GitHub wrote: Author: Godefroid Chapelle Author: Godefroid Chapelle Author: Godefroid Chapelle Author: Godefroid Chapelle Author: Godefroid Chapelle Author: Godefroid Chapelle I wish you hadn't pushed that -- some of these changes are definitely inappropriate on the 3.10 branch (adding an Acquisition dependency is definitely wrong). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLv87QACgkQ+gerLs4ltQ6XJQCgim/cILFy2rM0qpBO8/wUU0CP TDEAn0rnHd/KqejwVyeuTLiHesrmUDAR =E6G2 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB.FileStorage.format: TxnHeader cannot handle Unicode 'descr'
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/05/2013 01:13 PM, Jim Fulton wrote: > On Sat, Oct 5, 2013 at 1:47 AM, Chao Meng wrote: >> Hi there, >> >> Here is an issue from zopefoundation/ZODB github three months ago. I >> am facing similar issue now and don't know how to resolve. >> >> github issue link: https://github.com/zopefoundation/ZODB/issues/12 >> >> When I use Pyramid+ZODB+traversal and use some Chinese characters in >> URL. Note that my resource tree saved in ZODB with unicode fine for >> the Chinese characters as object names. >> >> Basically when save transaction, ZODB.FileStorage.format TxnHeader >> uses request.path_info as it's descr, which is unicode, but >> TxnHeader cannot handle Unicode :( >> >> It would be great if anyone can help or give some pointers. > > This is a Pyramid bug. > > transaction.note is defined to take a bytes string. Pyramid should > encode the path before passing it to transaction.note. The interfaces says "text". I realize that this is likely for "hysterical raisins", but if we mean "bytes", we should say so. Note that the implementation's use of an unadorned string literal to join the values means that in Py3k, it really *is* text, and not bytes. If we want the application to do the encoding, then we should change that literal as well. > Alternatively, Pyramid could store the path in transaction extended > info, which accepts any picklable type. > > Of course, we could revisit this. If we did, I'd deprecate the > transaction user and description attributes and only support meta data > via the extended info mechanism, which I'd rename "meta data". Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJS2g0ACgkQ+gerLs4ltQ5WEwCfdRPMCIrR5GdVaiPgLiBJT7QT 9wIAoLQzqhB9HRWYYIjsAx77+9A0dngc =PFfZ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZODB 4.0.0b3] DB.pack() broken?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 08/14/2013 02:05 PM, Tres Seaver wrote:
> On 08/14/2013 07:48 AM, [mabe] wrote:
>> Hi everyone,
>
>> I noticed today that database packing doesn't seem to work (at least
>> on Python 3.3)
>
>> Minimal example:
>
>> from ZODB import FileStorage, DB
>
>> storage = FileStorage.FileStorage('/tmp/mystorage.fs') db =
>> DB(storage) db.pack() db.close()
>
>> Expected: No output / db.pack() should succeed, although there is
>> likely nothing to pack.
>
>> Actual output: % python zodb_pack.py packing Traceback (most recent
>> call last): File "/usr/lib/python3.3/site-packages/ZODB/DB.py",
>> line 810, in pack self.storage.pack(t, self.references) File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
>> line 1121, in pack pack_result = self.packer(self, referencesf,
>> stop, gc) File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
>> line 1074, in packer opos = p.pack() File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 408, in pack self.gc.findReachable() File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 191, in findReachable self.findReachableAtPacktime([z64]) File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 276, in findReachableAtPacktime for oid in self.findrefs(pos): File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 329, in findrefs return self.referencesf(self._file.read(dh.plen))
>> File "/usr/lib/python3.3/site-packages/ZODB/serialize.py", line 646,
>> in referencesf u.noload() AttributeError: 'Unpickler' object has
>> no attribute 'noload' Traceback (most recent call last): File
>> "zodb_pack.py", line 5, in db.pack() File
>> "/usr/lib/python3.3/site-packages/ZODB/DB.py", line 810, in pack
>> self.storage.pack(t, self.references) File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
>> line 1121, in pack pack_result = self.packer(self, referencesf,
>> stop, gc) File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
>> line 1074, in packer opos = p.pack() File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 408, in pack self.gc.findReachable() File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 191, in findReachable self.findReachableAtPacktime([z64]) File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 276, in findReachableAtPacktime for oid in self.findrefs(pos): File
>> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
>> 329, in findrefs return self.referencesf(self._file.read(dh.plen))
>> File "/usr/lib/python3.3/site-packages/ZODB/serialize.py", line 646,
>> in referencesf u.noload() AttributeError: 'Unpickler' object has
>> no attribute 'noload'
>
> Thanks very much for the report. I can reproduce with your example
Christian Tismer submitted a pull request fixing the bug -- I verified,
merged it, and released zodbpickle 0.5.2 with the fix:
https://pypi.python.org/pypi/zodbpickle/0.5.2
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlIP87AACgkQ+gerLs4ltQ4wEACeIvdOrpmAGXvypMC4BVfl/80A
Fc4AniGouCvRK1wlV3Ds/Adx+B/yXo3Z
=OmFf
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbpickle: need a new tag
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 08/14/2013 02:14 PM, Tres Seaver wrote:
> On 07/16/2013 03:33 PM, Christian Tismer wrote:
>> Hi Stephan,
>
>> thanks a lot that you merged my work!
>
>> Now there is a little bit missing: For some reason, the 0.5.1 tag
>> was not merget from the clone. All files there, but the tag is
>> still 0.5.0.
>
>> I would like to push it on PyPI, but for that I need a new tag.
>
>> Can you please add a new tag, like 0.5.1 ?
>
>> Thanks & cheers -- chris
>
> Guys, the 0.5.1 releas / trunk has test failures under Py3k:
>
> https://github.com/zopefoundation/zodbpickle/issues/6
>
> In addition to the problem just reported ("Unpickler instance has no
> attribute 'noload'). In both cases, the 0.5.0 tag / release does not
> have the problem. Can you have a look?
Here is the issue for missing 'noload':
https://github.com/zopefoundation/zodbpickle/issues/7
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlIL0MIACgkQ+gerLs4ltQ4EUgCfWgaS9yUgy0+3cP8DNZbolxnz
RhcAn13hS2jnB44d1gfmvdD9RYtuNWa4
=0+eF
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbpickle: need a new tag
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 07/16/2013 03:33 PM, Christian Tismer wrote:
> Hi Stephan,
>
> thanks a lot that you merged my work!
>
> Now there is a little bit missing: For some reason, the 0.5.1 tag was
> not merget from the clone. All files there, but the tag is still
> 0.5.0.
>
> I would like to push it on PyPI, but for that I need a new tag.
>
> Can you please add a new tag, like 0.5.1 ?
>
> Thanks & cheers -- chris
Guys, the 0.5.1 releas / trunk has test failures under Py3k:
https://github.com/zopefoundation/zodbpickle/issues/6
In addition to the problem just reported ("Unpickler instance has no
attribute 'noload'). In both cases, the 0.5.0 tag / release does not
have the problem. Can you have a look?
Tres.
- --
=======
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlILyPEACgkQ+gerLs4ltQ4akQCbBTFaNf4JnLzFZ3m3s12DfxrO
TJYAnA5mDGyfbpeSL+jzFjJnMF3R50dJ
=Vd3B
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZODB 4.0.0b3] DB.pack() broken?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 08/14/2013 07:48 AM, [mabe] wrote:
> Hi everyone,
>
> I noticed today that database packing doesn't seem to work (at least
> on Python 3.3)
>
> Minimal example:
>
> from ZODB import FileStorage, DB
>
> storage = FileStorage.FileStorage('/tmp/mystorage.fs') db =
> DB(storage) db.pack() db.close()
>
> Expected: No output / db.pack() should succeed, although there is
> likely nothing to pack.
>
> Actual output: % python zodb_pack.py packing Traceback (most recent
> call last): File "/usr/lib/python3.3/site-packages/ZODB/DB.py", line
> 810, in pack self.storage.pack(t, self.references) File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
> line 1121, in pack pack_result = self.packer(self, referencesf, stop,
> gc) File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
> line 1074, in packer opos = p.pack() File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 408, in pack self.gc.findReachable() File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 191, in findReachable self.findReachableAtPacktime([z64]) File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 276, in findReachableAtPacktime for oid in self.findrefs(pos): File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 329, in findrefs return self.referencesf(self._file.read(dh.plen))
> File "/usr/lib/python3.3/site-packages/ZODB/serialize.py", line 646,
> in referencesf u.noload() AttributeError: 'Unpickler' object has no
> attribute 'noload' Traceback (most recent call last): File
> "zodb_pack.py", line 5, in db.pack() File
> "/usr/lib/python3.3/site-packages/ZODB/DB.py", line 810, in pack
> self.storage.pack(t, self.references) File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
> line 1121, in pack pack_result = self.packer(self, referencesf, stop,
> gc) File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py",
> line 1074, in packer opos = p.pack() File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 408, in pack self.gc.findReachable() File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 191, in findReachable self.findReachableAtPacktime([z64]) File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 276, in findReachableAtPacktime for oid in self.findrefs(pos): File
> "/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py", line
> 329, in findrefs return self.referencesf(self._file.read(dh.plen))
> File "/usr/lib/python3.3/site-packages/ZODB/serialize.py", line 646,
> in referencesf u.noload() AttributeError: 'Unpickler' object has no
> attribute 'noload'
Thanks very much for the report. I can reproduce with your example:
downgrading 'zodbpickle' to version 0.5.0 gets your example to run
cleanly, e.g.::
$ /path/to/virtualenv/bin/easy_install -U "zodbpickle == 0.5.0"
Could I get you to post it to the zodbpickle tracker:
https://github.com/zopefoundation/zodbpickle/issues
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlILxvoACgkQ+gerLs4ltQ5bXQCg2c7lb9Lu/179TuNpIQRFvhjN
1ecAoKhh+QaRiVnQXnO4MWu1Nyv4A+CE
=boEi
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] make ZODB as small and compact as expected
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/22/2013 09:15 AM, Stephan Richter wrote: >>> BTrees > I agree, this could be part of ZODB and it would be fine. Splitting out BTrees was a conscious decision to serve two goals: - - Allow evolving it (in particular, the work to port it to Py3k / PyPy) without stalling on the larger ZODB project. For ongoing work, it is useful to be able to release a fix for a BTrees-only bug without needing to release ZODB. - - Allow projects which use BTrees (as base classes or attributes) to be tested without needing to install all of ZODB. I consider both of those concerns still important, and so am -1 on re-absorbing BTrees into ZODB. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHtV3QACgkQ+gerLs4ltQ6PeACeNW11cnoU7IxG37tkOhGxuRXU RBEAmwXv+JD32emv2MFXaiAqmGcTj+Qt =oWDa -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbpickle claim (but OS X is not unix)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/04/2013 11:52 AM, Stephan Richter wrote: > On Thursday, July 04, 2013 05:47:39 AM Christian Tismer wrote: >> there is now a pull request, completely tested on OS X, ready to be >> applied with no other work involved. > > Yeah, I saw that. thanks. > >> It would be nice if that version could be uploaded, soon, so that I >> can close this issue and move on. ;-) > > Yeah, I just want Tres and/or Jim to weigh in, since they did the last > iterations on this code. And you know, we have Independence Day > today, so this week might be a little bit tough. ;-) I've done some review on Christian's PR: he is preparing some additional changes. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHVz5UACgkQ+gerLs4ltQ5ctwCeK79agF+YyKS/392i3aL9Xd87 7UoAn13Ii3C3UQqY2QoP6XFc3qPPt8Qu =9oCQ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Attempts at Python2-compatible pickles cause unpicklable objects
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/08/2013 02:03 AM, Tres Seaver wrote: > One point to consider is that anybody who has been using ZODB on Py3k > since the beta will have to do surgeery on their filestorages to get > them to load after that change (the code doesn't make any attempt to > determine if the magic number is later; it just checks for an exact > match). I don't want to make the "wait, we have five users, we can't > take out the tab requirement" mistake here: anybody for whom this > would be a hardship needs to speak up pretty quickly. I merged the 'issue_10' branch to trunk and released ZODB 4.0.0b3 yesterday. Today I found (as I expected) that I couldn't open the database I created last week under Py3k after updating. Am I the only person affected? Or do we need to add "fallback" code to permit opening a filestorage with an older magic number in read-only mode? In my case, the data is all test stuff, so I can just blow it away and recreate it. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG4rpcACgkQ+gerLs4ltQ4h/gCcCc/PBE6rY0FcJ2+pXqAAXpEJ SNkAn1v3O3ypcIfQYN+egYnBixNZIUii =zQ3t -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Attempts at Python2-compatible pickles cause unpicklable objects
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/07/2013 09:56 AM, Jim Fulton wrote: > On Fri, Jun 7, 2013 at 3:02 AM, Marius Gedminas > wrote: >> On Thu, Jun 06, 2013 at 09:43:16PM -0400, Stephan Richter wrote: >>> On Thursday, June 06, 2013 09:28:38 PM Tres Seaver wrote: >>>> If I am correct about this consensus, then in addition to >>>> merging PR #11 for ZODB, we should also drop the >>>> 'bytes_as_strings' changes to 'zodbpickle'. > > Yes > >>> >>> Yes, this is a left-over from our experimentation. It did not >>> work, since it also pickled attribute names as bytes and when you >>> loaded them, the attributes would be gone. >> >> The specific combination last used in ZODB4 on Python 3 (pickle >> bytes_as_strings=True, load with encoding='ascii', errors='bytes') >> handled attributes fine. >> >> Apparently it broke datetime objects -- and I never noticed. >> >>> Also, requiring protocol 3 for Python 3 sounds correct as well. >> >> How about also bumping the magic number of Data.fs files so ZODB on >> Python 2 would reject it early? I have made that change on the 'issue_10' branch. One point to consider is that anybody who has been using ZODB on Py3k since the beta will have to do surgeery on their filestorages to get them to load after that change (the code doesn't make any attempt to determine if the magic number is later; it just checks for an exact match). I don't want to make the "wait, we have five users, we can't take out the tab requirement" mistake here: anybody for whom this would be a hardship needs to speak up pretty quickly. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGyyT8ACgkQ+gerLs4ltQ4SxACgrskSD/t/9R0yFtlsSUQRGIuX 0REAoMWBctMgYYqgu0WiV2E5WCi790o4 =mazo -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RFC: Attempts at Python2-compatible pickles cause unpicklable objects
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I posted an issue, including a gist showing the problem: https://github.com/zopefoundation/ZODB/issues/10 The 'issue_10' branch adds a test (failing without the patch) and a patch for this issue: https://github.com/zopefoundation/ZODB/pull/11 I think we arrived at consensus that trying to emit backward-compatible pickles (bytes pickled using the string opcodes, as in Python2) was the wrong approach, and that when we get around to tackling the problem, we would do so by having Python2 emit forward-compatible pickles instead (using a custom 'binary' type for things which should be bytes in Py3k, and using the protocol 3 opcodes to pickle them). If I am correct about this consensus, then in addition to merging PR #11 for ZODB, we should also drop the 'bytes_as_strings' changes to 'zodbpickle'. Thoughts? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGxN0YACgkQ+gerLs4ltQ7+SgCeNLwvgq56afQnu06kw8bUO30E 69wAnR2WcFwz43N7QmxxBU93zxrcD/aO =/9eb -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/25/2013 01:43 PM, Tres Seaver wrote: > I will make a BTrees 4.0.7 release shortly. That would be 4.0.8: https://pypi.python.org/pypi/BTrees/4.0.8 Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGg+dsACgkQ+gerLs4ltQ6jTACgsMxUAD/oFW/h3MtcDw4wYWCU +NIAoJZOBLH4zGZwAbMBUjaSRZuOBJNb =NclA -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/25/2013 01:36 PM, Tres Seaver wrote: > I am trying to reproduce the ZEO test failure now, and then against > the BTrees trunk, before making a BTrees release. Rather than relying on the slow and somewhat non-deterministic ZEO failures, I have verified that the current tip of BTrees master permits Marius' 'zodbfail' tests to pass. I will make a BTrees 4.0.7 release shortly. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGg+FcACgkQ+gerLs4ltQ553wCggrDP4WCyWbxcd1LpEqts8fnZ gNgAnRwB/0oB8/psYdSVlXkFAV3GPv8P =q+w9 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/25/2013 11:49 AM, Tres Seaver wrote: > On 05/25/2013 02:50 AM, Marius Gedminas wrote: >> On Fri, May 24, 2013 at 09:31:55PM -0400, Tres Seaver wrote: >>> On 05/24/2013 01:28 PM, Marius Gedminas wrote: >>>> I was trying to debug a failing ZEO test case >>>> (testConcurrentUpdates), and found out it had nothing to do with >>>> ZEO. There's a bug either in BTrees or in ZODB itself that >>>> causes data corruption under Python 3.x. >>>> >>>> Here's a test case: >>>> https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py >>> >>> Thanks for the detective work! I can confirm the bug, and that >>> it is in the BTrees C extensions (the tests pass after 'rm >>> .tox/py32/lib/python3.2/site-packages/BTrees/*.so'). > >> It's the COMPARE macro in _compat.h. This patch fixes the data >> corruption: > >> diff --git a/BTrees/_compat.h b/BTrees/_compat.h index >> e004d54..4765df1 100644 --- a/BTrees/_compat.h +++ >> b/BTrees/_compat.h @@ -28,8 +28,8 @@ #define TEXT_FORMAT >> PyUnicode_Format > >> #define COMPARE(lhs, rhs) \ -PyObject_RichCompareBool((lhs), >> (rhs), Py_LT) > 0 ? -1 : \ -(PyObject_RichCompareBool((lhs), >> (rhs), Py_EQ) > 0 ? 0 : 1) +(PyObject_RichCompareBool((lhs), >> (rhs), Py_LT) > 0 ? -1 : \ + (PyObject_RichCompareBool((lhs), >> (rhs), Py_EQ) > 0 ? 0 : 1)) > > >> #else > > >> I'm now trying to write a failing unit test. It's a bit difficult >> to get access to C macros from Python tests ;-) > >> COMPARE is used in two places: > >> BTrees/objectkeymacros.h-#define TEST_KEY_SET_OR(V, KEY, TARGET) \ >> BTrees/objectkeymacros.h:if ( ( (V) = COMPARE((KEY),(TARGET)) ), >> PyErr_Occurred() ) > >> and > >> BTrees/objectvaluemacros.h:#define TEST_VALUE(VALUE, TARGET) >> COMPARE((VALUE),(TARGET)) > >> We can ignore TEST_KEY_SET_OR, as it works fine there. > >> TEST_VALUE is used to implement BTree.byValue and bucket merging. >> The bug I was chasing down was caused by bucket mismerges, I think. > >> It should be easier to write a test for byValue. In fact there is >> one, in test__base.py, but it's not run against real C-implemented >> BTrees. I copied it over into test_OOBTree.py and discovered that >> it fails unexpectedly: After adding your new test, and wrapping the 'COMPARE()' macro inside extra parens within 'TEST_VALUE' (in 'objectvaluemacros.h'), the new 'byValue' test passes on Py3k. I am trying to reproduce the ZEO test failure now, and then against the BTrees trunk, before making a BTrees release. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGg9rIACgkQ+gerLs4ltQ6M5ACeIyIgCC9/jWwew1qSFyJNg+eb EDIAnRORMJ2XN7xm8kjd+lZciEaJKkA0 =iXx3 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 05/25/2013 02:50 AM, Marius Gedminas wrote:
> On Fri, May 24, 2013 at 09:31:55PM -0400, Tres Seaver wrote:
>> On 05/24/2013 01:28 PM, Marius Gedminas wrote:
>>> I was trying to debug a failing ZEO test case
>>> (testConcurrentUpdates), and found out it had nothing to do with
>>> ZEO. There's a bug either in BTrees or in ZODB itself that
>>> causes data corruption under Python 3.x.
>>>
>>> Here's a test case:
>>> https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py
>>
>> Thanks for the detective work! I can confirm the bug, and that it
>> is in the BTrees C extensions (the tests pass after 'rm
>> .tox/py32/lib/python3.2/site-packages/BTrees/*.so').
>
> It's the COMPARE macro in _compat.h. This patch fixes the data
> corruption:
>
> diff --git a/BTrees/_compat.h b/BTrees/_compat.h index
> e004d54..4765df1 100644 --- a/BTrees/_compat.h +++ b/BTrees/_compat.h
> @@ -28,8 +28,8 @@ #define TEXT_FORMAT PyUnicode_Format
>
> #define COMPARE(lhs, rhs) \ -PyObject_RichCompareBool((lhs),
> (rhs), Py_LT) > 0 ? -1 : \ -(PyObject_RichCompareBool((lhs),
> (rhs), Py_EQ) > 0 ? 0 : 1) +(PyObject_RichCompareBool((lhs),
> (rhs), Py_LT) > 0 ? -1 : \ + (PyObject_RichCompareBool((lhs),
> (rhs), Py_EQ) > 0 ? 0 : 1))
>
>
> #else
>
>
> I'm now trying to write a failing unit test. It's a bit difficult to
> get access to C macros from Python tests ;-)
>
> COMPARE is used in two places:
>
> BTrees/objectkeymacros.h-#define TEST_KEY_SET_OR(V, KEY, TARGET) \
> BTrees/objectkeymacros.h:if ( ( (V) = COMPARE((KEY),(TARGET)) ),
> PyErr_Occurred() )
>
> and
>
> BTrees/objectvaluemacros.h:#define TEST_VALUE(VALUE, TARGET)
> COMPARE((VALUE),(TARGET))
>
> We can ignore TEST_KEY_SET_OR, as it works fine there.
>
> TEST_VALUE is used to implement BTree.byValue and bucket merging.
> The bug I was chasing down was caused by bucket mismerges, I think.
>
> It should be easier to write a test for byValue. In fact there is
> one, in test__base.py, but it's not run against real C-implemented
> BTrees. I copied it over into test_OOBTree.py and discovered that it
> fails unexpectedly:
>
> diff --git a/BTrees/tests/test_OOBTree.py
> b/BTrees/tests/test_OOBTree.py index bc046bc..e813f88 100644 ---
> a/BTrees/tests/test_OOBTree.py +++ b/BTrees/tests/test_OOBTree.py @@
> -99,9 +99,15 @@ class OOSetPyTest(ExtendedSetTests,
> unittest.TestCase):
>
> class OOBTreeTest(BTreeTests, unittest.TestCase):
>
> -def _makeOne(self): +def _makeOne(self, *args): from
> BTrees.OOBTree import OOBTree -return OOBTree() +
> return OOBTree(*args) + +def test_byValue(self): +ITEMS =
> [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] +
> tree = self._makeOne(ITEMS) +
> self.assertEqual(list(tree.byValue(22)), +
> [(y, x) for x, y in ITEMS[22:]])
>
> def testRejectDefaultComparison(self): # Check that passing int keys w
> default comparison fails. @@ -146,9 +152,9 @@ class
> OOBTreePyTest(OOBTreeTest): # default 'object' comparison semantics.
> #class OOBTreePyTest(BTreeTests, unittest.TestCase):
>
> -def _makeOne(self): +def _makeOne(self, *args): from
> BTrees.OOBTree import OOBTreePy -return OOBTreePy() +
> return OOBTreePy(*args)
>
>
>
>
> The failure is
>
> AssertionError: Lists differ: [(25, 'z'), (24, 'y'), (23, 'x... !=
> [(22, 'w'), (23, 'x'), (24, 'y...
>
> First differing element 0: (25, 'z') (22, 'w')
>
> - [(25, 'z'), (24, 'y'), (23, 'x'), (22, 'w')] + [(22, 'w'), (23,
> 'x'), (24, 'y'), (25, 'z')]
>
> so the C-implemented BTrees return values in descending order, while
> the base Python implementation returns them in ascending order. This
> happens on Python 2.7 as well, so it's not related to the COMPARE
> macro bug.
>
> Does anybody know what the API of BTree.byValue() is supposed to be?
> The interface just says "Return a sequence of (value, key) pairs,
> sorted by value", without clarifying if it means an ascending or a
> descending sort.
The docstring in the C module says:
byValue(min) -> list of value, key pairs
Returns list of value, key pairs where the value is >= min. The
list is sorted by value. Note that items() returns keys in the
opposite order.
So I guess the descending order is intentional: the call to '.reversed'
is certainly there. Let's upd
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 05/24/2013 01:28 PM, Marius Gedminas wrote:
> I was trying to debug a failing ZEO test case
> (testConcurrentUpdates), and found out it had nothing to do with ZEO.
> There's a bug either in BTrees or in ZODB itself that causes data
> corruption under Python 3.x.
>
> Here's a test case:
> https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py
>
> It initializes an OOBTree to {0:0, 1:0, 2:0}, then launches two
> threads that attempt just one write each:
>
> tree[1] = 1 # thread 1 tree[2] = 2 # thread 2
>
> All transactions are committed successfully and the end result
> _sometimes_ is {0:0, 1:0, 2:2} instead of the expected {0:0, 1:1,
> 2:2}.
>
> There's also a larger test case (zodbfail.py) that modifies more than
> one item in a thread. That one fails pretty reliably.
>
> This only happens under Python 3.x.
>
> To reproduce::
>
> git clone git://gist.github.com/5644876.git zodbfail cd zodbfail
> detox
Thanks for the detective work! I can confirm the bug, and that it is in
the BTrees C extensions (the tests pass after 'rm
.tox/py32/lib/python3.2/site-packages/BTrees/*.so').
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlGgFIsACgkQ+gerLs4ltQ5QxgCgves8VGndIxt37UCV/CPMKeuw
t3EAoK+gwFE3SNOpBSLJ0p+mZ4ftP7gl
=wxp6
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/10/2013 07:55 PM, Stephan Richter wrote: > On Friday, May 10, 2013 05:04:31 PM Tres Seaver wrote: >> I pushed out a ZODB 4.0.0b1 release after the merge. If the >> buildbots stay green over the weekend, I think we can release a >> 4.0.0 final early next week. > > Awesome thanks. After the 4.0.0 final release, I will set aside some > time for the CH devs to release final versions of all the ZTK > packages. > > Tres, are we ready to commit to a zope.security 4.0.0 as well? AFAIK, we could cut it from the 'master' at any time. I don't know of any issues I don't see any open launchpad issues which should block a release. tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGNtIUACgkQ+gerLs4ltQ6mkACffp0He4ybnf0dkTJwLCiJxtjy b7EAn0S6iQX6sgroQ66VFfFMXeyQOZob =eY19 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/08/2013 12:34 PM, Tres Seaver wrote: > On 04/29/2013 08:37 PM, Stephan Richter wrote: >> Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready >> for Py3 with noload() support. I totally agree that we do not need >> to solve any of the transition work now. > >> So for ZODB Py3 support we need to: > >> 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just >> contain the cPickle code that is Py3 compatible. > >> I do not care whether this happens for ZODB 4.0 or 4.1 as long as I >> get some commitment that 4.1 > > Chris and I chatted with Jim about this over beers last Friday. I > explained that the current 'py3; branch does not require the > 'zodbpickle everywhere' stuff (the Python2 side doesn't use > 'zodbpickle'). Jim then agreed that we could merge that branch before > releasing 4.0. We will need to add some caveats to the docs / > changelog (Python3 support is only for new applications, no forward- / > backward-compatibility for data, etc.) > > Given that ZODB won't import or use 'zodbpickle' under Python2, I > don't think we need to remove the current Python2 support (as released > in 0.4.1): the Python3 version (with noload()) has been there all > along. I have merged the 'py3' branch to 'master': - - All tests pass under all four platforms using buildout. - - All unit tests pass on all four platforms using 'setup.py test'. I added the following note to the changelog: ZODB 4.0.x is supported on Python 3.x for *new* applications only. Due to changes in the standard library's pickle support, the Python3 support does **not** provide forward- or backward-compatibility at the data level with Python2. A future version of ZODB may add such support. Applications which need migrate data from Python2 to Python3 should plan to script this migration using separte databases, e.g. via a "dump-and-reload" approach, or by providing explicit fix-ups of the pickled values as transactions are copied between storages. I pushed out a ZODB 4.0.0b1 release after the merge. If the buildbots stay green over the weekend, I think we can release a 4.0.0 final early next week. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGNYN8ACgkQ+gerLs4ltQ5GPACaA3EtwgZOUsvWmsoGOTWi0bpw n5kAoNNUVfsofX/eKxFZKd3F/pLQvvDG =7oxn -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 08:37 PM, Stephan Richter wrote: > Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready > for Py3 with noload() support. I totally agree that we do not need to > solve any of the transition work now. > > So for ZODB Py3 support we need to: > > 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just > contain the cPickle code that is Py3 compatible. > > I do not care whether this happens for ZODB 4.0 or 4.1 as long as I > get some commitment that 4.1 Chris and I chatted with Jim about this over beers last Friday. I explained that the current 'py3; branch does not require the 'zodbpickle everywhere' stuff (the Python2 side doesn't use 'zodbpickle'). Jim then agreed that we could merge that branch before releasing 4.0. We will need to add some caveats to the docs / changelog (Python3 support is only for new applications, no forward- / backward-compatibility for data, etc.) Given that ZODB won't import or use 'zodbpickle' under Python2, I don't think we need to remove the current Python2 support (as released in 0.4.1): the Python3 version (with noload()) has been there all along. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGKfngACgkQ+gerLs4ltQ500gCfQK4HSzemxaYkcPAyleNdkagq MAwAn3wYoCo4BItBHAve4o+lhrzRTBrt =BbCU -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 12:44 PM, Jim Fulton wrote: > Yes. There are lots of features I'd like to add to ZODB. I tend to > work on them when I have time (infrequently) or where we have a > driving need at ZC. Long ZODB release cycles provide a lot of stop > energy. We are already developing this way (the 'py3' branch has not been merged). However, if you do a lot of Python2-only feature work and merge to master, you will likely push back to horizon for merging that branch: we will have to port any work done to it. Using the "4.0" label to signal "big changes ahead, evaluate carefully before upgrading" was the primary reason I had been pushing to get the Py3k stuff landed (the low-risk thing would have been more naturally labeled "3.11"). FWIW, we have reports that some brave souls have successfully built Py3k apps using the 'py3' branch. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+qrEACgkQ+gerLs4ltQ5OpgCg1FZ8xB9EijUQOQqzhm2RAZ4V +z4An29kZZYpoFlFeu0QMpLbqGRfIscV =5EKv -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 11:00 AM, Jim Fulton wrote: > Let's keep master stable. Maybe someone will want to add features > before the Python 3 support is stable. I don't want to hold 4.1 > hostage either. Given that the only folks (besides maybe you) invested in ZODB development want Py3k support ASAP. I don't see that. Do you have features in mind that you would imagine releasing before we land Py3k support? > I suggest breaking the Python 3 work into increments that can each be > introduced without sacrificing stability. > > The first increment could provide Python 3 support without any > conversion or compatibility support. This is something you could > probably achieve pretty quickly and would allow you meet your > immediate goals. We are already there, AFAIK, on the 'py3' branch: the blocker is just getting out a release. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+nugACgkQ+gerLs4ltQ4lmACgmfAMS0tqV86+v0ItIlbMkhtK i6gAoLPO4aehpHIrZokPG8c4hRIAOsnE =6f/j -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 10:51 AM, Jim Fulton wrote: > Right. As I suggested, let's get to a point where we can get a stable > ZODB 4.0 release for Python 2. As soon as we get that, let's get a > ZODB 4.0.x or 4.1 release that works on Python 3, presumably via > zodbpickle. As I proposed earlier this morning, we can make a non-Py3k, non-zodbpickle 4.0b1 release today from the master branch, and a 4.0 final in a week. Once we get that release out, we can then merge the 'py3' branch, including adding the requirement for 'zodbpickle' under both Python2 and Py3k, and aim for a much expedited 4.1 release which supports Py3k. At this point, I would favor putting the database conversion / straddling code into a new package (not 'zodbpickle'). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+iZ4ACgkQ+gerLs4ltQ7SgACfW0UMVtp1MJGplWDEBfx3CM9V pesAoNqece3ZQ/EVoFad6G2+B3vXkCz8 =ig9W -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 09:48 AM, Jim Fulton wrote: > On Sun, Apr 28, 2013 at 8:34 PM, Stephan Richter > wrote: >> On Sunday, April 28, 2013 07:23:12 PM Jim Fulton wrote: >>> Can ZODB 4 be used now without zodbpickle? >> >> No, unfortunately for Py2 we need the custom cPickle and for Py3 >> `noload()` support (as Tres mentioned). > > This is a problem. > > The only change in ZODB 4.0 was supposed to be the breakup. > > This was supposed to be a low-risk release. The separation into > multiple packages was supposed to increase agility, but now it appears > we're stuck. The only reason we had delayed the 4.0 release (in my mind, anyway) was that it was a good way to signal the Py3k compatibliity changes. I'm not wedded to calling the Py3k-compatible release "4.0". > I'd like there to a stable 4.0 release **soon** that doesn't use > zodbpickle for Python 2. > > For now, I suggest we focus on stability and the ability to make > progress on non-Python-3-related work. > > After that is achieved, I suggest we get to the point where people > can create new databases and use them with Python 3. We need to do > this without hindering the ability to make new stable releases. The trunk of the 'ZODB' package does not have any of the Py3k / zodbpickle changes yet. We could make a ZODB 4.0b1 release from the trunk today and create a '4.0' stable branch prior to any merge of the 'py3' work. > As far as the grander vision for Python2/3 transition and > interoperability, we need to make progress incrementally and not > sacrifice stability of the master branch. > > I made the 3.11 release fully expecting a stable 4.0 release soon. That was of the 'ZODB3' meta-package, right? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+gboACgkQ+gerLs4ltQ4+7wCg222VrN5b0jkRrSJKVBL1VEBr 5lgAoINrzLbTus6ycBXcVGovxWIPBQ5t =XLcz -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/28/2013 07:19 PM, Jim Fulton wrote: > I'm confused. I don't understand why we need a Python 3 pickler > change to support the new Python 2 binary type. I thought we were > going to pickle Python 2 binary objects using the standard Python 3 > protocol 3 code? We need 'noload' support on Py3k. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF9sc0ACgkQ+gerLs4ltQ40kwCfeH++IWblTJRNDyma/OZQkGX6 tYwAmQFCs5ynIDzONiBKbqC//sD0MdUk =0xq4 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 04:38 PM, Tres Seaver wrote: > - ``zodbpickle`` should provide a new ``binary`` type which Python2 > applications can begin using to signal that attributes should be > unpickled in Py3k as ``bytes``. See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - ``zodbpickle`` should provide a pickler/unpickler for use by Python2 > clients who operate against converted storages (replace_py2_cpickle_). > See: > https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes > > - ``zodbpickle`` should provide a pickler/unpickler for use by Py3k > clients who operate against unconverted storages > (replace_py3k_pickle_). See: > https://github.com/zopefoundation/zodbpickle I have pushed a new branch, 'merge_py2_py3k', which basically holds the existing Py3k forks (from master) alongside the Pyton-2 forks (from 'py2_explicit_bytes'): https://github.com/zopefoundation/zodbpickle/tree/merge_py2_py3k After thrashing and failing in my attempt to have the Python code straddle, I instead have separate versions (based on Python 2.7 and 3.2), and a unifying 'zodbpickle.pickle' module which pulls in the standard API from the appropriate module. This solution is ugly, but it does allow for running tests cleanly on each supported version of Python. It also minimized the extent of patching for each fork (the straddled version I discarded was a franken-module by comparison). I would like to merge this branch to master early next week and make a release, so that we can evaluate merging the 'py3' branch of ZODB. Thoughts? Note that I have not yet addressed the portions of my proposal which deal with analyzing / converting existing databases, or with the possibly-needed wrapper storage (for on-the-fly conversion). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF68tcACgkQ+gerLs4ltQ4lzQCdHvsnCZ5cfZlycJ1BQw8Uct5q CGYAnilMitegoqsVOevpPlbLcuu7bur6 =qEsj -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Which version is the current version of ZODB?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/26/2013 11:51 AM, Matthew Kasfeldt wrote: > Hi I am new to the community and not sure where else to ask this. I > am learning ZODB and see that there is a ZODB3 and ZODB4. I am unsure > of which one is the current stable release of the product or if they > are both current but used for different purposes. I on the ZODB site > it lists ZODB3 and on wikipedia it states the most current stable is > 3.10.3. But looking at various repositories (GitHub and pyppi) I see > version 4 is also available (and a 3.11.0a3 alpha release as well). I > currently use Python 2.7x and want to be sure I am using the most > current stable code to learn on. Any input on the matter would be > greatly appreciated! The 3.10.x line is the current stable release line: you want to be installing 'ZODB3==3.10.5'. ZODB 4.x is not yet released: we are hoping to finish the Py3k support soon and make a beta. The code is actually still quite mature, with very good test coverage: it should be stable enough for ordinary development, although I wouldn't want to put it into production just yet. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF6uKwACgkQ+gerLs4ltQ52HgCfU3i5BzzYuFzep0Zl9ZzSh4G7 QQIAn3sxTZIcCe+ZtmQRi7Q0pCq6soJA =F6rH -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 05:19 PM, Stephan Richter wrote: > Oh, I see that in your branch of ``zodbpickle`` you already added > protocol 3 support to the Py2.7 cPickle code. Does this code also run > under Py3.3? (I don't now why Stephan's e-mail didn't make it to the list). I would not expect to use the Python 2.7-derived pickler on Py3k. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFu7+UACgkQ+gerLs4ltQ4EWgCgyGHjPlqWEK5+PFhJXK94jCLY edMAn2aZV5dLxV7XdTgSh83yOBrdnTL/ =/tPT -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 05:13 PM, Stephan Richter wrote: > On Tuesday, April 16, 2013 04:38:06 PM Tres Seaver wrote: >> Comments? (I don't now why Stephan's e-mail didn't make it to the list). > The big omission that I noticed while reading the text carefully is a > note saying that you will never be able to use stock Py3k pickle, > because it does not support noload(). Thus ``zodbpickle`` is needed > for any Py3k code. (I think this is a correction to you your last > bullet in _replace_py2_cPickle.) Hmm, I think you are correct. > That reminds me, originally we forked pickle.py from Python 3.3. > During PyCon I think you decided to start by using cPickle from Python > 2.7 instead. If you are starting from Py2.7 cPickle, then supporting > Protocol 3 is not easy. Already done (as you note in your follow-up). > Given your writeup, I think you are implicitly saying to start from > Py3.3 pickle and add the special support for Python 2 binary via the > special new type. That sounds good to me. I would actually prefer to fork the Python 3.2 version: the one from 3.3 pulls in a bunch of grotty internal-only usage. > BTW, what are your motivations for all the different strategies? I wanted to document them all, because some of the strategies suit different cases better than others. > _ignore_compat is obvious. If you can easily create the ZODB from > other data sources, then you can do a one-time switch. In fact, at > CipherHealth we have this case, since the ZODB only contains config > (which is loaded from text files) and session data. Yup. Even for large CMS systems, I would still make "dump-to-filesystem, then reload" a requirement. Others disagree, of course (and may have legitimate reasons). Leo Rochael Almeida has clients with databases "too big to convert", for instance (the downtime required to do the conversion would be prohibitive, I believe). > But which strategy would be useful for a large Plone site for example? > I think we should focus on that and provide one good way to do it. Plone has historically preferred in-place migration to dump-reload. Maybe jumping the Py3k curb is enough reason for them to reconsider. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFu79sACgkQ+gerLs4ltQ4aTQCfSeb6Xiz0OOtoGJuzSKfOetMu 7IAAoMiNzaohY2lv8DMoLRmoNIw8f3P0 =fcYz -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RFC: Python2 - Py3k database compatibility
k; however, *all* bytes data will be coerced (via ``latin1``) to unicode. - - Pickles created under Py3k will likely not be readable on Python2 (Python2 has no support for ``protocol 3``). - - Easiest usage for applications which are never going to straddle. - - Compatibility will only be achievalble via one-time conversions (where the conversion script uses one of the other strategies or tools). .. _replace_py3k_pickle: Replace Py3k ``pickle`` +++ Keep pickling in the Python2 / protocol 1 way we have always done. - - No changes to the ``ZODB`` packages on Python2. Storages do not need to be configured with any custom pickle support. - - On Py3k, ``ZODB`` uses pickler / unpickler from the ``zodbpickle`` module, such that Python2 ``str`` objects are unpickled as ``bytes``; ``bytes`` are pickled using the ``protocol 1`` opcodes (so that Python2 will unpickle them as ``str``). .. _replace_py2_cPickle: Replace Python2 ``cPickle`` +++ Move to pickling in the new protocol 3 way (native under Py3k). - - On Python2, applications which need to ensure that ``bytes`` objects unpickle correctly under Py3k need must be changed to use a new type, ``zodbpickle,binary``. ``ZODB`` is configured with pickler / upickler from ``zodbpickle``, such that objects of this type will be pickled using the ``protocol 3`` opcodes for bytes (so that Py3k will unpickle them as ``bytes``). - - Existing data for the affected classes will need to be fixed up using a variation of convert_storages_. - - No changes to the ``ZODB`` packages on Py3k. Storages do not need to be configured with any custom pickle support. .. _convert_storages: Convert Database Storages + - - Need tool(s) to identify problematic data: - Classes which mix ``str`` and ``unicode`` values for the same attribute across records / instances. - - Utility which can apply per-class transforms to state pickles: - E.g., for instances of ``OFS.Image.Pdata``, convert the ``data`` attribute (which should be a Python2 ``str``) to ``zodbpickle.binary``. (Of course, these would probably be better off written out as blobs). - Or, for some application which mixes ``str`` and ``unicode`` under Python2 (either across instances or across transaction): upconvert any value of type ``str`` for the given attribute(s) to ``unicode``, using a configured encoding strategy (e.g, try ``utf8`` first, falling back to ``latin1``). - - One-time converter utility would use ``copyTransactionsFrom``-style pattern, opening the existing database readonly, getting pickles for each transaction, invoking the converter utility for each instance to fix up the pickle, then writing the converted pickles into the new database. .. _wrap_storages: Wrap Database Storages ++ - - A wrapper storage uses the converter utility (identified above) during the ``load`` operation, fixing up the object state it is handed to the instance's ``__setstate__``. - - During the ``save`` operation, the wrapper would fix up pickled instance state (after calling ``__getstate__``). - - Wrappers might be applied under Python2 (e.g., for apps where the databse is already converted to ``protocol 3``) as an alternative to replace_py2_cpickle_. - - Wrappers might be applied under Py3k (e.g., for apps where the databse is not already converted to ``protocol 3``) as an alternative to replace_py3k_pickle_.. Concrete Proposal - - I believe we will need to update ``zodbpickle`` and ``ZDOB`` to allow for any of the strategies to be applied. - - ``zodbpickle`` should provide the script which analyzes pickles in a database for inconsistent ``str`` / ``unicode`` usage. See: https://github.com/jimfulton/dbstringanalysis - - ``zodbpickle`` should provide the utility for registering per-class fixups. - - ``zodbpickle`` should provide the script which uses that utility do to one-time conversion of a storage (supporting convert_storages_). - - ``zodbpickle`` should provide a new ``binary`` type which Python2 applications can begin using to signal that attributes should be unpickled in Py3k as ``bytes``. See: https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes - - ``zodbpickle`` should provide a pickler/unpickler for use by Python2 clients who operate against converted storages (replace_py2_cpickle_). See: https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes - - ``zodbpickle`` should provide a pickler/unpickler for use by Py3k clients who operate against unconverted storages (replace_py3k_pickle_). See: https://github.com/zopefoundation/zodbpickle - - ``zodbpickle`` might need to provide a wrapper storage supporting straddle_no_convert_. Comments? Tres. - -- === Tres Seaver +1 5
Re: [ZODB-Dev] Towards ZODB on Python 3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 03/10/2013 11:55 AM, Jim Fulton wrote:
> On Sun, Mar 10, 2013 at 11:25 AM, Tres Seaver
> wrote:
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
>>
>> On 03/10/2013 09:19 AM, Jim Fulton wrote:
> ...
>>> I think the fix is pretty straightforward.
>>>
>>> In the default __setstate__ provided by Persistent, and when
>>> loading non-persistent instances:
>>>
>>> - On Python 2, ASCII encode unicode attribute names.
>>>
>>> - On Python 3, ASCII decode byte attribute names.
>>>
>>> The same transformation is necessary when looking up global
>>> names.
>>
>> Hmm, if zodbpickle has to handle the issue for non-persistent
>> instances and global names, wouldn't it be simpler to make it handle
>> persistent instances too?
>
> No. It can't know when a key is going to be used for a persistent
> attribute name.
>
>> It can examine the stack inside 'load_dict' to figure out that the
>> context is an instance, right?
>
> Ugh. What stack?
The one where the unpickler keeps its work-in-progress?
static int
load_none(UnpicklerObject *self)
{
PDATA_APPEND(self->stack, Py_None, -1);
return 0;
}
static int
load_dict(UnpicklerObject *self)
{
PyObject *dict, *key, *value;
Py_ssize_t i, j, k;
if ((i = marker(self)) < 0)
return -1;
j = Py_SIZE(self->stack);
if ((dict = PyDict_New()) == NULL)
return -1;
for (k = i + 1; k < j; k += 2) {
key = self->stack->data[k - 1];
value = self->stack->data[k];
if (PyDict_SetItem(dict, key, value) < 0) {
Py_DECREF(dict);
return -1;
}
}
Pdata_clear(self->stack, i);
PDATA_PUSH(self->stack, dict, -1);
return 0;
}
> It would be much simpler to handle this in __setstate__ (or the
> equivalent). This isn't exactly a lot of code.
Again, if we have to deal with non-persistent / global names inside the
pickler anyway, then why not do the same for persistent ones?
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlE8sSoACgkQ+gerLs4ltQ7veQCg1wIhHOo93Z3z3NTVG3tesJnS
GLAAoL5XNrzOR5oucmPcpDSNE3QJ7Kb9
=YUjX
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Towards ZODB on Python 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/10/2013 09:19 AM, Jim Fulton wrote: >>> Now, ZODB is built on top of pickles. And pickles in Python 2 >>> know about two kinds of strings: str and unicode. But there are >>> actually *three* kinds of strings in Python-land: >>> >>> * bytes * unicode * native strings (same as bytes in Python 2, >>> same as unicode in Python 3) > I hadn't encountered that term before. I see it informally used to > refer to ``str``, which is bytes in Python 2 and Unicode in Python 3. > This isn't a different kind of string. WSGI (PEP ) uses the term. In the case of pickles, we need to guarantee that the strings used for attribute names (including module attributes) are native strings. Oops, I see that you arrive there below: I should have read all the way to the bottom. > Is this an issue for anything but names (object attributes and global > names)? > > I don't think there's a "native strings" issue. There *does* seem to > be an name issue. In Python 2 and Python 3, (non-buggy) unicode > aware applications use bytes and unicode the same way, unicode for > text, bytes for data. > > AFAICT, Python 3 has (admirably) changed the way names are > implemented to use unicode, rather than ASCII. > > Am I missing something? > > This is a somewhat thorny, but still fairly restricted problem. I > would hazard to guess that 99.923% of persistent classes pickle their > state using their instance dictionaries. 99.9968% for regular Python > classes. We know when we're pickling and unpickling instances and we > can apply transformations necessary for the target platforms. > > I think the fix is pretty straightforward. > > In the default __setstate__ provided by Persistent, and when loading > non-persistent instances: > > - On Python 2, ASCII encode unicode attribute names. > > - On Python 3, ASCII decode byte attribute names. > > The same transformation is necessary when looking up global names. Hmm, if zodbpickle has to handle the issue for non-persistent instances and global names, wouldn't it be simpler to make it handle persistent instances too? It can examine the stack inside 'load_dict' to figure out that the context is an instance, right? > This will cover the vast majority of cases where the default > __setstate__ is used. In rare cases where a custom setstate is used, > or when Python 3 non-ASCII attribute names are used, then databases > may not be sharable accross Python versions. Code with custom __setstate__ / __getstate__ where the difference matters is going to need porting anyway, so it might as well straddle. > There is also likely to be breakage in dictionaries or BTrees where > applications are sloppy about mixing Unicode and byte keys. I don't > think we should try to compensate for this. Mixing None into a tree with bytes / text keys is likely a bigger problem than mixed bytes / text keys. > These applications need to be fixed. One could write a database > analysis script to detect this kind of breakage (looking for mixed > string and unicide keys). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEUEARECAAYFAlE8pdoACgkQ+gerLs4ltQ7EswCeLZ0qq3INcYIxVNI55d6wurgy d5UAlA1jwoI1tWFwxfVp2ZhrO92Ctyk= =rPzb -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [zopefoundation/zodbpickle] 661e8b: Add minimal tests and Python ref. impl. for 'noloa...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/06/2013 06:29 AM, Marius Gedminas wrote: > On Tue, Mar 05, 2013 at 05:05:36PM -0800, GitHub wrote: >> Branch: refs/heads/master Home: >> https://github.com/zopefoundation/zodbpickle Commit: >> 661e8b78f91a9e77da739c2f1d00780ad24d1efa >> https://github.com/zopefoundation/zodbpickle/commit/661e8b78f91a9e77da739c2f1d00780ad24d1efa >> >> Author: Tres Seaver >> Date: 2013-03-05 (Tue, 05 Mar 2013) >> >> Changed paths: M src/zodbpickle/pickle.py M >> src/zodbpickle/tests/pickletester.py >> >> Log Message: --- Add minimal tests and Python ref. impl. for >> 'noload'. > > Thank you, this made my day! > > BTW I've a plan for Python 3.2 compatibility in zodbpickle: move the > new features (noload, bytes_as_strings) into _pickle_32.c, rename it > to _pickle.c, then drop _pickle_33.c and the corresponding tests for > 3.3-only features. > > We're not using any of the new pickle features introduced in Python > 3.3, and I don't like to see an external package like zodbpickle > using undocumented internal CPython APIs like _Py_IDENTIFIER. Sounds good to me. Note that the test coverage for the Python reference implementation is not great (and not just for 'noload*'). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlE3z4UACgkQ+gerLs4ltQ565wCgsk8K0qMqGA3FHUqXx3F5G09s 5D0AoJZep0kkI/S4mvxf8eBY9g62C5Tk =AIxH -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB Py3k porting status
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thanks to Marius and Stephan's efforts (with a little help from me), the 'py3' branch of ZODB now passes all unit tests, with most of the many-many ResourceWarnings now suppressed as well (we are down to the last two unclosed blob files). Today, I worked on making the branch work with Python 3.2. In order to support 3.2, I needed a 3.2-compatible verison of 'zodbpickle', as well as ZConfig. The ZConfig changes were trivial (avoiding u'' literals): I pushed out a 3.0.3 release with those changes. For 'zodbpickle', I created a second fork of '_pickle.c' (from the Python 3.2.3 release), and select between the two variants at build time based on the version of Python. I couldn't make a release there (Marius is the sole PyPI owner / maintainer), but was able to install it in my ZODB checkout's '.tox/py32' virtualenv for testing. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEyp/kACgkQ+gerLs4ltQ7rowCeOsClMleFcDRFAYiNjaz6tyaJ wR4An2otbJ7c5KpRiV36jX24S6fJpADd =i/tT -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] API question
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 01/15/2013 06:11 AM, Jim Fulton wrote:
> On Mon, Jan 14, 2013 at 1:32 PM, Tres Seaver
> wrote:
>> -BEGIN PGP SIGNED MESSAGE- Hash: SHA1
>>
>> While working on preparation for a Py3k port, I've stumbled across
>> a fundamental issue with how ZODB structures its API. Do we intend
>> that client code do the following::
>>
>> from ZDOB import DB, FileStorage db =
>> DB(FileStorage('/path/to/Data.fs'))
>
> As Marius points out, this doesn't work.
>
>>
>> or use the module as a facade ::
>>
>> import ZODB db =
>> ZODB.DB(ZODB.FileStorage.FileStorage('/path/to/Data.fs'))
>
> This doesn't work either. You haven't imported FileStorage.
>
> WRT ZODB.DB, ZODB.DB is an age-old convenience. It's unfortunate that
> ZODB.DB (the class) shadows the module, ZODB.DB, just like the class
> ZODB.FileStorage.FileStorage shadows the modules
> ZODB.FileStorage.FileStorage.FileStorage. (Of course, it's also
> unfortunate that there's a ZODB.FileStorage.FileStorage.FileStorage
> module. :)
>
> If we had a do-over, we'd use ZODB.db.DB and
> ZODB.filestorage.FileStorage, and ZODB.DB would be a convenience for
> ZODB.db.DB.
>
>>
>> I would actually prefer that clients explicitly import the
>> intermediate modules::
>>
>> from ZDOB import DB, FileStorage db =
>> DB.DB(FileStorage.FileStorage('/path/to/Data.fs'))
>
> So you don't mind shadowing FileStorage.FileStorage.FileStorage. ;)
>
>> or even better::
>>
>> from ZDOB.DB import DB # This one can even be ambiguous now
>
> FTR, I don't like this style. Somewhat a matter of taste.
Sure. I don't like using APIs via long, multi-dotted paths.
>> from ZODB.FileStorage import FileStorage db =
>> DB(FileStorage('/path/to/Data.fs'))
>>
>> The driver for the question is getting the tests to pass under both
>> 'nosetests' and 'setup.py test', where the order of module imports
>> etc. can make the ambiguous cases problematic. It would be a good
>> time to do whatever BBB stuff we need to (I would guess figuring out
>> how to emit deprecation warnings for whichever variants) before
>> releasing 4.0.0.
>
> I'm pretty happy with the Zope test runner and I don't think using
> nosetests is a good reason to cause backward-incompatibility. The
> zope test runner works just fine with Python 3. Why do you feel
> compelled to introduce nose?
I have a couple of goals:
- - I want 'python setup.py develop test' to work on a fresh checkout, mostly
because it makes doing multi-Python tests with tox simple.
- - Nose gives us easy access to Ned Batchelder's coverage package, which
allows me to get seriously good test coverage in place before trying to
port code to the "straddle" dialect.
- - I mean to keep the buildout + testrunner stuff working, too.
> I'm sort of in favor of moving to nose to follow the crowd, although
> otherwise, nose is far too implicit for my taste. It doesn't hande
> doctest well at all.
>
> Having said that, if I was going to do something like this, I'd rename
> the modules, ZODB.DB and ZODB.FileStorage to ZODB.db and
> ZODB.filestorage and add module aliases for backward compatibility. I
> don't know if that would be enough to satisfy nose.
Likely so. I will give it a stab on the branch I'm working with.
> I'm not up for doing any of this for 4.0. I'm not alergic to a 5.0
> in the not too distant future. I'm guessing that a switch to nose
> would also make you rewrite all of the doctests as unittests. As the
> prrimary maintainer of ZODB, I'm -0.8 on that.
I'd like to see ZODB ported to Py3k ASAP, in a single codebase (as with
the ported 'zope.*' packages, 'persistent', 'transaction', and 'BTrees').
Trying to get there with doctests has seemed way too hard to me:
doctests are really fragile to cross-platform changes (especially various
'repr' changes).
In the case of the already-ported packages, I migrated most of their
doctests out of the software into testsed snippets inside Sphinx docs.
That move seems like a reasonable tradeoff: the examples still get
exercised as part of the docs, but they don't carry the weight of testing
the package.
> Back to APIs, I think 90% of users don't import the APIs but set up
> ZODB via ZConfig (or probably should, if they don't). For Python
> use, I think the ZODB.DB class short-cut us useful. Over the last
> few years, ZODB has grown some additional
Re: [ZODB-Dev] API question
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/14/2013 03:33 PM, Marius Gedminas wrote: > Can you demonstrate the ambiguity? As I've shown before, I was > unable to find it, at least with Python 2.x. On the filesystem: $ find src/ZODB/FileStorage/ -name "*.py" src/ZODB/FileStorage/fspack.py src/ZODB/FileStorage/__init__.py src/ZODB/FileStorage/fsdump.py src/ZODB/FileStorage/fsoids.py src/ZODB/FileStorage/format.py src/ZODB/FileStorage/interfaces.py src/ZODB/FileStorage/tests.py src/ZODB/FileStorage/FileStorage.py So, 'ZODB.FileStorage.FilesStorage' could logically be either the 'Filestorage' module inside the 'ZODB.FileStorage' package, or else the same-named class. As with ZODB.DB, getting to the actual module is tricky, because both thses imports give you the class:: from ZODB.FileStorage import FileStorage and:: From ZODB.FileStorage.FileStorage import FileStorage I'm tempted to rename the 'DB.py' module 'db.py', and jam in a BBB entry in sys.modules for 'ZODB.DB'; likewise, I am tempted to rename the 'FileStorage.py' package 'filestorage', its same-named module '_filestorage.py', and jam in BBB entries for the old names. Those renames would make the "preferred" API: >>> from ZODB import DB # convenience alias for the class >>> from ZODB import db # the moodule >>> from ZODB.db import DB # my preferred speling >>> from ZDOB.filestorage imoprt FileStorage # conv. alias for class >>> from ZODB import filestorage # the package >>> from ZODB.filestorage import FileStorage # my preferred speling >>> from ZODB.filestorage import _filestorage # if needed For extra bonus fun, we could rename 'ZODB' to 'zodb' :) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlD0oOgACgkQ+gerLs4ltQ4LOwCgu3VSRklLjFMdkuWLkUNV4h2S m/MAoKMI+ZrTqFUnXkgGNSw7Gq2yYN0V =67De -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] API question
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
While working on preparation for a Py3k port, I've stumbled across a
fundamental issue with how ZODB structures its API. Do we intend that
client code do the following::
from ZDOB import DB, FileStorage
db = DB(FileStorage('/path/to/Data.fs'))
or use the module as a facade ::
import ZODB
db = ZODB.DB(ZODB.FileStorage.FileStorage('/path/to/Data.fs'))
I would actually prefer that clients explicitly import the intermediate
modules::
from ZDOB import DB, FileStorage
db = DB.DB(FileStorage.FileStorage('/path/to/Data.fs'))
or even better::
from ZDOB.DB import DB
# This one can even be ambiguous now
from ZODB.FileStorage import FileStorage
db = DB(FileStorage('/path/to/Data.fs'))
The driver for the question is getting the tests to pass under both
'nosetests' and 'setup.py test', where the order of module imports etc.
can make the ambiguous cases problematic. It would be a good time to do
whatever BBB stuff we need to (I would guess figuring out how to emit
deprecation warnings for whichever variants) before releasing 4.0.0.
Tres.
- --
=======
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlD0TycACgkQ+gerLs4ltQ4AgACg3MCYrEOga5KF8goWyu2OxjWe
H7QAoLEyHTShzBc9ZkMENWbG+hqzrpTg
=nBoy
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Moving ZEO, BTrees, persistent and transaction to github
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/04/2013 01:19 PM, Jim Fulton wrote: > On Fri, Jan 4, 2013 at 1:05 PM, Jim Fulton wrote: >> On Wed, Jan 2, 2013 at 9:37 AM, Jim Fulton wrote: >>> On Fri, Dec 28, 2012 at 4:44 PM, Jim Fulton wrote: >>>> I plan to do ZEO shortly. >>> >>> Well, that didn't go well. svn git fetch spent several days and >>> didn't finish. It seemed to be really thrashing trying to follow >>> tags. >>> >>> I'm probably going to have to convert just the trunk. >>> >>> I'm guessing that this is related to the project split. >>> >>>> I'll do and BTrees, persistent and transaction in about a week. >>> >>> Or so. :) I suspect that these will have the same problem and >>> that I'll have to convert just the trunk. >> >> persistent went smoothly enough. persistent is now in github. > > BTrees and transaction converted smoothly too. Go figure. > > This weekend, I'll update svn for these packages and ZODB to indicate > that development is taking place in github. The persistent repo on Github is missing three revisions I made yesterday, as well as the 4.0.6 tag. Patch attached (the tag should be made against the repository before applying the last revision). The 'BTrees' and 'transsaction' repos are missing the empty 'docs/_static' and 'docs/_templates' directories, which causes Sphinx to emit a warning. I have forked them and added pull requests which put them back using placeholders: https://github.com/zopefoundation/transaction/pull/1 https://github.com/zopefoundation/BTrees/pull/1 I would have just pushed these changes, but have only read-only access to those repositories. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlDnPY8ACgkQ+gerLs4ltQ7ddQCbB80wW0J5ngL071Q/T4GGOk/L Nr4AoMz/B5NAoe5EFDuGJmXFacCcANfK =RhVS -END PGP SIGNATURE- revno: 140 committer: Tres Seaver branch nick: persistent timestamp: Thu 2013-01-03 16:41:17 -0500 message: svb diff: === modified file 'CHANGES.txt' --- CHANGES.txt 2013-01-03 21:39:44 + +++ CHANGES.txt 2013-01-03 21:41:17 + @@ -2,6 +2,10 @@ +4.0.7 (unreleased) +-- + + 4.0.6 (2013-01-03) -- === modified file 'setup.py' --- setup.py 2013-01-03 21:39:44 + +++ setup.py 2013-01-03 21:41:17 + @@ -12,7 +12,7 @@ # ## -__version__ = '4.0.6' +__version__ = '4.0.7dev' import os import platform revno: 139 committer: Tres Seaver branch nick: persistent timestamp: Thu 2013-01-03 16:39:44 -0500 message: Prep 4.0.6 release. diff: === modified file 'CHANGES.txt' --- CHANGES.txt 2013-01-03 21:39:27 + +++ CHANGES.txt 2013-01-03 21:39:44 + @@ -2,7 +2,7 @@ -4.0.6 (unreleased) +4.0.6 (2013-01-03) -- - Updated Trove classifiers. === modified file 'setup.py' --- setup.py 2013-01-03 21:39:27 + +++ setup.py 2013-01-03 21:39:44 + @@ -12,7 +12,7 @@ # ## -__version__ = '4.0.6dev' +__version__ = '4.0.6' import os import platform revno: 138 committer: Tres Seaver branch nick: persistent timestamp: Thu 2013-01-03 16:39:27 -0500 message: Update Trove classifiers. diff: === modified file 'CHANGES.txt' --- CHANGES.txt 2012-12-14 15:49:15 + +++ CHANGES.txt 2013-01-03 21:39:27 + @@ -5,7 +5,7 @@ 4.0.6 (unreleased) -- -- TBD +- Updated Trove classifiers. 4.0.5 (2012-12-14) === modified file 'setup.py' --- setup.py 2012-12-14 15:49:15 + +++ setup.py 2013-01-03 21:39:27 + @@ -80,6 +80,7 @@ 'Programming Language :: Python :: 3.3', "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", +"Framework :: ZODB", "Topic :: Database", "Topic :: Software Development :: Libraries :: Python Modules", "Operating System :: Microsoft :: Windows", ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Progress report: porting persistent, BTrees to Python3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 12/13/2012 09:40 PM, Tres Seaver wrote:
> I have gotten the trunk of persistent to build an test persistent's
> extension modules under Python 3.2. There is a segfault under Python
> 3.3 which looks to be an actual Python bug:
>
> http://bugs.python.org/issue16676
>
> I will make a 4.0.5 release once I figure out what to do about that
> test under 3.3 (maybe I'll just skip it).
I fixed the remainig issues in persistent and released 4.0.5 today: its
tests properly exercise the C extensions Under Python 3.2 / 3.3.
> On the BTrees front: I have the extension modules compiling under
> Python3 (the 'py3k' branch), but didn't read Lennart's excellent
> guide closely enough to do the new module initialization dance
> correctly. I plan to fix that tomoorow and hope to have the C
> versions of the trees passing all the same tests as the pure-python
> implementations. Once they pass, I will merge that branch to the
> trunk and look to cut a new release.
>
> Both packages are set up to test against all supported Python
> versions using tox: at this point, that includes 2.6, 2.7, PyPy, 3.2,
> and 3.3. The tox config also builds Sphinx docs and exercises the
> doctest snippets, as well as running coverage (at this point only
> under Python2).
I'm down to one test failure (out of 3600+) under Py3k: the test that
inserting an out-of-bounds long long key into an L*BTree. On Python 3.2,
the assertion fails::
==
FAIL: test64 (BTrees.tests.testBTrees.FamilyTest)
- --
Traceback (most recent call last):
File
"/home/tseaver/projects/Zope/ZODB/BTrees-py3k/BTrees/tests/testBTrees.py", line
481, in test64
self.assertRaises(ValueError, s.insert, BTrees.family64.maxint + 1)
AssertionError: ValueError not raised by insert
On Python 3.3, the test fails with an OverflowError::
==
ERROR: test64 (BTrees.tests.testBTrees.FamilyTest)
- --
Traceback (most recent call last):
File
"/home/tseaver/projects/Zope/ZODB/BTrees-py3k/BTrees/tests/testBTrees.py", line
481, in test64
self.assertRaises(ValueError, s.insert, BTrees.family64.maxint + 1)
File "/opt/Python-3.3.0/lib/python3.3/unittest/case.py", line 571, in
assertRaises
return context.handle('assertRaises', callableObj, args, kwargs)
File "/opt/Python-3.3.0/lib/python3.3/unittest/case.py", line 135, in
handle
callable_obj(*args, **kwargs)
File "/opt/Python-3.3.0/lib/python3.3/unittest/case.py", line 153, in
__exit__
self.obj_name))
OverflowError: Python int too large to convert to C long
The latter one is easy, I think: we could just have the test assert
either ValueError *or* OverflowError. Figuring out why the insert
doesn't raise at all under 3.2 is too hard for my bleary eyes to tackle
tonight (I've already reverted one failed stab at it).
One good thing: the tests under Py3k now take about thes ame time as the
CPython 2.x ones (just over a minute on my box): that is down from over
five minutes before getting the C extensions working. The PyPy tests,
running without extensions, take about three minutes.
Onward-and-upward'ly,
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlDLybgACgkQ+gerLs4ltQ6VSwCgmwLLIEOArUzSsOLbMWnRhq2Q
a04AoIWVjuWqkYrSOo/3ukNL7rAiKskY
=yLOx
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Progress report: porting persistent, BTrees to Python3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have gotten the trunk of persistent to build an test persistent's extension modules under Python 3.2. There is a segfault under Python 3.3 which looks to be an actual Python bug: http://bugs.python.org/issue16676 I will make a 4.0.5 release once I figure out what to do about that test under 3.3 (maybe I'll just skip it). On the BTrees front: I have the extension modules compiling under Python3 (the 'py3k' branch), but didn't read Lennart's excellent guide closely enough to do the new module initialization dance correctly. I plan to fix that tomoorow and hope to have the C versions of the trees passing all the same tests as the pure-python implementations. Once they pass, I will merge that branch to the trunk and look to cut a new release. Both packages are set up to test against all supported Python versions using tox: at this point, that includes 2.6, 2.7, PyPy, 3.2, and 3.3. The tox config also builds Sphinx docs and exercises the doctest snippets, as well as running coverage (at this point only under Python2). Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlDKkYwACgkQ+gerLs4ltQ5kcgCgqY8h7yLohuw4ukiYpiO0eeHZ HaQAnjVKDr4YcQtF+KKVD7tn/v+Lx+8f =T1HG -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: persistent/trunk/ Worked around test failure due to overflow to long on 32-bit systems.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 11/15/2012 11:47 AM, Jim Fulton wrote:
> On Sun, Aug 26, 2012 at 2:30 PM, Tres Seaver
> wrote:
>> Log message for revision 127583:
>
> ...
>
>> Modified: persistent/trunk/persistent/cPersistence.c
>> ===
>
> ...
>
>> @@ -1118,6 +1118,14 @@ { if (v) { + if (PyLong_Check(v)) +
>> { + long long llv = PyInt_AsLongLong(v); + if (llv
>> > sys_maxint) + { + v = sys_maxint; /* borrow
>> reference */ + } + }
>
> This is wrong. You're comparing a long long and a PyObject*.
Thanks. The attached patch should fix it to compare the value of the
largest allowed size, rather than its pointer. Does that look correct?
>> Modified: persistent/trunk/persistent/tests/test_persistence.py
>> ===
>> --- persistent/trunk/persistent/tests/test_persistence.py
>> 2012-08-26 16:19:07 UTC (rev 127582) +++
>> persistent/trunk/persistent/tests/test_persistence.py
>> 2012-08-26 18:30:20 UTC (rev 127583) @@ -504,9 +504,18 @@
>>
>> def test_assign_p_estimated_size_bigger(self): inst =
>> self._makeOne() -inst._p_estimated_size = 1073741697 * 1024
>> +inst._p_estimated_size = 1073741697 * 4 #still <= 32 bits
>> self.assertEqual(inst._p_estimated_size, 16777215 * 64)
>
> This test is meant to test that the code that prevents us from
> overflowing the 30-bits reserved for storring estimated sizes. Using
> * 2 (rather than * 4) should be enough.
OK. If we are truncating anyway, it shouldn't matter.
> We don't really need to support Python longs, do we? We just need to
> fix the test to not provoke a long.
>
> If someone sets a size >2G on a 32-bit platform, they'll get a type
> error. I don't think this is a big problem. If they're doing this,
> they probably have other problems. :)
The C code raises ValueError in that case. I don't care whether we
truncate or not: if you think that an exception works, I'm fine with
changing the tests (and the Python implementation) to fit.
> I propose to revert this change and change 1024 to 2.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlClKGAACgkQ+gerLs4ltQ7xGwCggARtQgeOSrAES6QJplyer+/k
a44AoJQZoEMzyrGoTrAzCf256EkKSOsv
=ogkb
-END PGP SIGNATURE-
=== modified file 'persistent/cPersistence.c'
--- persistent/cPersistence.c 2012-08-26 18:58:51 +
+++ persistent/cPersistence.c 2012-11-15 17:33:30 +
@@ -24,7 +24,8 @@
};
/* These two objects are initialized when the module is loaded */
-static PyObject *TimeStamp, *py_simple_new, *sys_maxint;
+static PyObject *TimeStamp, *py_simple_new, *max_est_size;
+static int max_est_size_i = 1073741696;
/* Strings initialized by init_strings() below. */
static PyObject *py_keys, *py_setstate, *py___dict__, *py_timeTime;
@@ -1121,9 +1122,9 @@
if (PyLong_Check(v))
{
long long llv = PyLong_AsLongLong(v);
- if (llv > sys_maxint)
+ if (llv > max_est_size_i)
{
- v = sys_maxint; /* borrow reference */
+ v = max_est_size; /* borrow reference */
}
}
if (PyInt_Check(v))
@@ -1434,14 +1435,10 @@
return;
}
- if (!sys_maxint)
+ if (!max_est_size)
{
- m = PyImport_ImportModule("sys");
- if (!m)
-return;
- sys_maxint = PyObject_GetAttrString(m, "maxint");
- Py_DECREF(m);
- if (!sys_maxint)
+ max_est_size = PyInt_FromInt(max_est_size_i);
+ if (!max_est_size)
return;
}
}
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Call for volunteers: help w finishing Python BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/09/2012 10:17 AM, Leonardo Santagada wrote: > On Fri, Nov 9, 2012 at 12:55 PM, Tres Seaver > wrote: > >> On 11/09/2012 09:33 AM, Leonardo Santagada wrote: >>> Great! Do you already have a benchmark suite to test its >>> performance on pypy? I think the next step would be to run ZODB on >>> pypy and measure performance, or is there more code that needs >>> porting? >> >> I'd like to finish cleanups, including 100% coverage, and think >> about Py3k porting for BTrees. We may need to refactor the >> pure-Python bits somewhat to allow better JITting by PyPy, too -- >> likely taht would need some help from them? >> > > The pypy guys would probably like to help zodb, one thing that they > keep repeating is that to help any project they need a self running > benchmark, something you can just run and then analise the results. If > you guys have it I'm betting the pypy team can help to make it fast. > My tip would be to call it the zope/plone database benchmark so that > it would carry more the weight of the killer apps using it (that have > a good market share in the python world). Shane's "ZODB shootout" buildout would probably be the best benchmark. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCdjcIACgkQ+gerLs4ltQ7zUwCgq6TqWdy601tnLxDEhCCK0rNi X/AAnRJRXXipGC7SG1CbQfMwOcao9iXK =TbFK -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Call for volunteers: help w finishing Python BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/21/2012 06:50 PM, Tres Seaver wrote: > On 10/04/2011 01:32 PM, Jim Fulton wrote: >> On Tue, Oct 4, 2011 at 11:36 AM, David Glick >> wrote: >>> On 10/4/11 8:33 AM, Jim Fulton wrote: >>>> >>>> Someone recently told me I should be more agressive about asking >>>> for help. >>>> >>>> If someone is looking for an opportunity to help, finishing the >>>> Python version of BTrees would help a lot. I think I got this >>>> started pretty well, but ran out of time. This is needed for >>>> running ZODB on PyPy and jython, both of which I'd like to see. >>>> >>>> svn+ssh://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees >>>> >>>> >>>> >>>> Jim >>>> >>>> P.S. Much thanks to Tres for his work on the Python version of >>>> persistence. >>> >>> What tasks remain to be done? (I assume running the tests will >>> give a starting point, but perhaps there are other todo items you >>> know of?) > >> Really, just getting the tests to pass. I think there are a lot of >> legacy, but still supporte features that need to be fixed. (This >> is a really old package.) > > In a fresh checkout of the branch, I see what looks like an infinite > loop in the tests: I left it running for an hour just now, and it > hung inside the '_set_operation' helper function inside the > 'test_difference' testcase for 'PureOO' testcase. Just a quick update: my 'pure_python' branch now passes all tests on Python 2.6, 2.7, and PyPy (no C extension1) I plan to do a lot of cleanup during the PyConCA sprints next week before merging the branch to the trunk. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCcf7gACgkQ+gerLs4ltQ5E8wCfdvUsjKmFvhsnZcKTVqbZa1Qz eKoAoNED3rKprPxqJxjToxN2txLgz9nN =a1x+ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 4.0.0a1 released
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/08/2012 05:24 PM, Alex Clark wrote: > On 2012-11-08 14:16:48 +, Jim Fulton said: > >> On Thu, Nov 8, 2012 at 4:21 AM, Marius Gedminas >> wrote: >>> On Wed, Nov 07, 2012 at 04:45:59PM -0500, Jim Fulton wrote: >>>> Not to be confused with ZODB3! :) >>>> >>>> This is the first ZODB 4 release. >>> >>> Not to be confused with the old ZODB4 either, I suppose ;) >>> >>> http://wiki.zope.org/ZODB/ZODB4 >> >> How could anyone be confused by that? ;) > > > Congrats on the release! FWIW, IIRC: "pip install ZODB" failed while > easy_install succeeded. Not sure why. Yeah, just tried it again: > > > - error: can't copy 'include/persistent/cPersistence.h': doesn't exist > Interesting that easy_install doesn't barf. We don't any longer ship headers or C code in the ZODB, so that piece of the setup() should go. Patch attached. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCcXJkACgkQ+gerLs4ltQ4VIACfTR6Hey7c5kjJVA+MLEtU6Mzb 4WMAnAyn7dHA8EEtzYHQ6nlypaRETGrc =BHhe -END PGP SIGNATURE- Index: setup.py === --- setup.py (revision 128202) +++ setup.py (working copy) @@ -121,10 +121,6 @@ test_suite="__main__.alltests", # to support "setup.py test" tests_require = tests_require, extras_require = dict(test=tests_require), - # XXX: We don't really want to install these headers; we would - # prefer just including them so that folks can build from an sdist. - headers = ['include/persistent/cPersistence.h', - 'include/persistent/ring.h'], install_requires = [ 'transaction', 'persistent', ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/20/2012 01:47 PM, Jim Fulton wrote: I had the impression that Tres was proposing more. I released BTrees 4.0.0, and created a ZODB branch for the (trivial) shift to depending on it: http://svn.zope.org/ZODB/branches/tseaver-btrees_as_egg/ That branch passes all tests, and should be ready for merging. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCC/WIACgkQ+gerLs4ltQ4c8QCeNSG1owrfJy1WRU3R9OEGTVBi Zx8An0PEnU0kpeYL1RkBlkQ5jfycw3vo =rDzU -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/14/2012 06:23 PM, Jim Fulton wrote: > On Sun, Oct 14, 2012 at 6:07 PM, Laurence Rowe wrote: >> On 14 October 2012 22:49, Jim Fulton wrote: >>> On Sun, Oct 14, 2012 at 5:28 PM, Tres Seaver >>> wrote: ... >>>>> Well, I don't have time to chase BTrees. This could always >>>>> be done in ZODB 5. :) >>>> >>>> I could help chop BTrees out, if that would be useful: most of >>>> the effort will be purely subtractive in the ZODB package (I >>>> don't think anything depends on BTrees). >>> >>> FileStorage uses BTrees for it's in-memory index. >>> >>> MappingStorage used BTrees. >>> >>> There are ZODB tests that use BTrees, but I suppose they could be >>> fixed. >>> >>> I just don't think the win is that great in separating BTrees at >>> this time. >> >> I don't think Hanno is suggesting removing BTrees as a dependency >> from ZODB but rather breaking out the BTrees package into a >> separate PyPI distribution to make it more visible to potential >> users outside of the ZODB community, e.g. >> http://www.reddit.com/r/Python/comments/exj74/btree_c_extension_module_for_python_alpha/ > >> >> > I had the impression that Tres was proposing more. Nope. I just meant to break BTrees out and have ZODB depend on it. I thought is would be helpful to decouple the release process a bit, e.g. so that the "pure-python BTrees" stuff could move forward independently of the bigger ZODB release cycle. >> To do that, refactoring tests shouldn't be required. I guess it >> could be argued that the fsBTree should be part of the ZODB rather >> than BTrees distribution, but leaving it where it is would be much >> easier. > > Yup. > > If someone wants to do this in the next couple of days, it would be > welcome. I really want to get these releases going. (And I have > limited time.) I have split BTrees out to a separately-releasable project: http://svn.zope.org/BTrees/trunk/ If going forward with that is OK, I would be glad to do the work to: - Release BTrees - Update ZODB trunk to depend on it by the end of the weekend. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCBcqUACgkQ+gerLs4ltQ7peACcD/MaZw5p26Ac5S3mp0EO3gpd c1gAoMrfgkqxh+ouy0Y22ERHFa2ONDB3 =K6VQ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/14/2012 05:20 PM, Jim Fulton wrote: > On Sun, Oct 14, 2012 at 5:03 PM, Hanno Schlichting > wrote: >> On Sun, Oct 14, 2012 at 10:21 PM, Jim Fulton wrote: >>> Goal: Give persistent, ZODB and ZEO their own release cycles. >> >> In which distribution would BTrees end up in? > > Still ZODB. > >> I think the pure-Python work for BTrees isn't yet finished, but I >> could be wrong. > > Not sure. > >> But if we are extracting packages into separate distributions, we >> should move BTrees out as well. > > Well, I don't have time to chase BTrees. This could always be done in > ZODB 5. :) I could help chop BTrees out, if that would be useful: most of the effort will be purely subtractive in the ZODB package (I don't think anything depends on BTrees). > BTW, I expect to increase the tempo of ZODB and ZEO releases. Yay. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlB7LpAACgkQ+gerLs4ltQ6QbACgkLzap7zb/n0Bcc/b82AkYfG/ nPkAoIzF6jFO7lvr9lAN8dgLeIlN3ack =AZ3M -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/27/2012 09:55 AM, Hanno Schlichting wrote: > On Mon, Aug 27, 2012 at 3:18 PM, Tres Seaver > wrote: >> I think that was the attempt to convert a too-big number to a C >> 'long long'. I have adjusted the test to use 2**63 - 1 directly. > > Yep, tests pass here now. > >> I will make a 4.0.2 release after to buildbots report success on >> Windows (assuming they do). > > The winbot only builds actual tagged releases. You can follow its > progress at http://winbot.zope.org/builders/wineggbuilder - there's a > run every 30 minutes. > > I'm confident enough in the fix, so I'd just cut a new release. OK, I'll make the release then. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA7h+kACgkQ+gerLs4ltQ7ALwCfY7nVKv1ZzXvYM3iduSwzCa37 iaQAn3xfaOnQfc+ZLkoT6lmQtPUPeMMM =eZGj -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/26/2012 07:05 AM, Hanno Schlichting wrote: > On Sat, Aug 25, 2012 at 4:11 PM, Tres Seaver > wrote: >> This failure, and the others like it, indicate that the buildout >> has failed to install persistent correctly (there is nothing >> Windows-specific about the failure):: > > I actually got the same error locally. > > The trouble is the timestamp.py module and the TimeStamp.c extension. > During installation setuptools creates a wrapper for the extension > file. So you end up with TimeStamp.so but also a timestamp.py > wrapper. This wrapper overwrites the original timestamp.py module. > > You likely have a case-sensitive file system and therefor didn't get > this problem. Whoever came up with the idea of a case-insensitive filesystem should be shot (or at least, whoever decided to keep doing that after moving off of DOS). I have worked around this problem (I think) by renaming the extension module to '_timestamp.so'. I released a 4.0.1 to PyPI with this fix, along with a fix for the int -> long overflow on 32-bit systems. > I think TimeStamp.c has been there before and is imported in other > modules as "from persistent.TimeStamp import TimeStamp". So we should > keep that API and use a different name for the new timestamp.py > module. I don't see the need for the BBB-preservation here: I documented the backward-incompatibility in CHANGES.txt instead. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA6c9IACgkQ+gerLs4ltQ6ocACeIqlGWazwos+yWGaMvgMCOkQD ZQoAn3oR2rY70a1oT6aTjLpofIVtVBNd =GzPs -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/25/2012 04:33 AM, Adam GROSZER wrote: > Hello, > > I'm sorry to say, but the build (for r127209) before your merge was > passing. > > See http://localhost:8009/builders/ZODB_dev%20py_265_win32/builds/750 This failure, and the others like it, indicate that the buildout has failed to install persistent correctly (there is nothing Windows-specific about the failure):: - - %< --- Error in test test___delattr___p__names (persistent.tests.test_persistence.PyPersistentTests) Traceback (most recent call last): File "c:\Python26_32\lib\unittest.py", line 279, in run testMethod() File "c:\eggs\persistent-4.0.0-py2.6-win32.egg\persistent\tests\test_persistence.py", line 643, in test___delattr___p__names inst, jar, OID = self._makeOneWithJar() File "c:\eggs\persistent-4.0.0-py2.6-win32.egg\persistent\tests\test_persistence.py", line 58, in _makeOneWithJar from persistent.timestamp import _makeOctets ImportError: cannot import name _makeOctets - - %< --- The only solution I can think of is to delete those buildouts and start fresh. > And there are more problems: > > http://winbot.zope.org/builders/persistent_py_265_32/builds/2/steps/test/logs/stdio > > all probably windows related. I don't have a clue how to diagnose this one. $ python 2.6 Python 2.6.8 (unknown, May 27 2012, 16:15:53) [GCC 4.6.3] on linux3 ... >>> type(1073741697 * 1024) What does that yield on the Windows box? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA43PUACgkQ+gerLs4ltQ4UrQCgzUc2GiwTd+FLu+UEJvCTeJSC mO4AoKy6rZvWHIVR5l9Cu6cpbLE5ajaP =Kgjg -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/24/2012 10:05 AM, Tres Seaver wrote: > On 08/24/2012 02:42 AM, Adam GROSZER wrote: >> Hello, >> >> I think this broke ZODB tests. >> >> http://winbot.zope.org/builders/ZODB_dev%20py_265_win64/builds/757/steps/test/logs/stdio >> >> >> >> >> >> >> On 08/21/2012 07:08 PM, Tres Seaver wrote: >>> Log message for revision 127545: Note split of persistent. >>> >>> Changed: _U ZODB/trunk/ U ZODB/trunk/src/CHANGES.txt >>> >>> -=- Modified: ZODB/trunk/src/CHANGES.txt >>> === >>> --- ZODB/trunk/src/CHANGES.txt 2012-08-21 17:07:01 UTC (rev >>> 127544) +++ ZODB/trunk/src/CHANGES.txt 2012-08-21 17:08:42 UTC >>> (rev 127545) @@ -8,6 +8,9 @@ New Features >>> >>> +- The ``persistent`` package is now released as a separate >>> distribution, + on which ZODB now depends. + - ZODB no longer >>> depends on zope.event. It now uses ZODB.event, which uses >>> zope.event if it is installed. You can override ZODB.event.notify >>> to provide your own event handling, although > > Hmmm, those symbols are there: > > $ /opt/Python-2.6.8/bin/virtualenv /tmp/per4 ... $ > /tmp/per4/bin/easy_install persistent ... $ /tmp/per4/bin/python -c > "from persistent.timestamp import _makeOctets" > > I will try a fresh ZODB checkout and see if I can reproduce. I don't see any of those failures. I see only one failure in src/ZEO/tests/testZEO.py (ZEO.tests.testZEO.client_has_newer_data_than_server). It looks like a spurious doctest failure to me (the expected error is actually raised, but the rendered output doesn't match). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA3jsIACgkQ+gerLs4ltQ6vtACglapHfxTy0FV/V8fpTKkNcGpZ 0HYAnR9lKiNkc2yG1XjuJU9+DECvvtlz =V8FU -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Call for volunteers: help w finishing Python BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/04/2011 01:32 PM, Jim Fulton wrote: > On Tue, Oct 4, 2011 at 11:36 AM, David Glick > wrote: >> On 10/4/11 8:33 AM, Jim Fulton wrote: >>> >>> Someone recently told me I should be more agressive about asking >>> for help. >>> >>> If someone is looking for an opportunity to help, finishing the >>> Python version of BTrees would help a lot. I think I got this >>> started pretty well, but ran out of time. This is needed for >>> running ZODB on PyPy and jython, both of which I'd like to see. >>> >>> svn+ssh://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees >>> >>> Jim >>> >>> P.S. Much thanks to Tres for his work on the Python version of >>> persistence. >> >> What tasks remain to be done? (I assume running the tests will give >> a starting point, but perhaps there are other todo items you know >> of?) > > Really, just getting the tests to pass. I think there are a lot of > legacy, but still supporte features that need to be fixed. (This is a > really old package.) In a fresh checkout of the branch, I see what looks like an infinite loop in the tests: I left it running for an hour just now, and it hung inside the '_set_operation' helper function inside the 'test_difference' testcase for 'PureOO' testcase. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA0EKQACgkQ+gerLs4ltQ5XtwCfZ5LKg214I/I5/s3Pd+AtpgzI Ur0AnRIvmkBvuZKfZtlDg9mv1wObnUwX =Ocn+ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: release persistent as a standalone package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/10/2012 08:52 AM, Tres Seaver wrote: > On 06/30/2012 02:02 PM, Tres Seaver wrote: > >> I would like to release a '4.0.0' version of the package, and >> switch the ZODB trunk to pull it in as a dependency (deleting the >> currently included (older) copy of persistent). One possible issue >> is that I have not (yet) made the C extensions work under Python >> 3.2: I don't know whether that should be a blocker for a release. > > I have a branch of ZODB which makes this go: > > http://svn.zope.org/ZODB/branches/tseaver-persistent_as_egg/ > > (note that the branch assumes 'persistent' can be installed from PyPI, > or that you have separately installed it beforehand). > > In order to work around that fact that setuptools / distribute do not > install headers when it installs eggs[1][2], I have tweaked the > setup.py to pass a "lazy string" for the 'include_path' entry to allow > BTrees to find the 'persisent/cPersistence.h' header. > > [1] http://bugs.python.org/setuptools/issue142 [2] > https://bitbucket.org/tarek/distribute/issue/295 I released persistent 4.0.0 today to PyPI: http://pypi.python.org/pypi/persistent and updated my ZODB branch to use it http://svn.zope.org/ZODB/branches/tseaver-persistent_as_egg/?rev=127487&view=rev At this point, I believe my branch is ready to merge to the trunk. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAmn70ACgkQ+gerLs4ltQ63qwCgqoG5ZE/X8HCgnlYuCU89OXEb XsgAn1PGyqY0or+6ifFvjvH40KMgp6g2 =lncl -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: release persistent as a standalone package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/30/2012 02:02 PM, Tres Seaver wrote: > I would like to release a '4.0.0' version of the package, and switch > the ZODB trunk to pull it in as a dependency (deleting the currently > included (older) copy of persistent). One possible issue is that I > have not (yet) made the C extensions work under Python 3.2: I don't > know whether that should be a blocker for a release. I have a branch of ZODB which makes this go: http://svn.zope.org/ZODB/branches/tseaver-persistent_as_egg/ (note that the branch assumes 'persistent' can be installed from PyPI, or that you have separately installed it beforehand). In order to work around that fact that setuptools / distribute do not install headers when it installs eggs[1][2], I have tweaked the setup.py to pass a "lazy string" for the 'include_path' entry to allow BTrees to find the 'persisent/cPersistence.h' header. [1] http://bugs.python.org/setuptools/issue142 [2] https://bitbucket.org/tarek/distribute/issue/295 Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/8JXcACgkQ+gerLs4ltQ6OVwCeIR6w7qOgXMlddKfdKi46TOqC CIYAoK2kGDm45wTTjNn/fIp6JTATk5fL =k0wr -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RFC: release persistent as a standalone package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have completed the work needed to make 'persistent' distributable as a standalone package. The effort (begun almost four years ago!) includes the following highlights: Platform Changes - - - Added explicit support for Python 3.2 and PyPy. - Note that the C implementations of Persistent, PickleCache, and Timestamp are not built (yet) on these platforms. - - Dropped explicit support for Python < 2.6. - - Completed pure-Python reference implementations of 'Persistent', 'PickleCache', and 'TimeStamp'. Testing Changes - --- - - 100% unit test coverage. - - All covered platforms tested under 'tox'. - - Refactored some doctests as unittests. - - Removed all 'ZODB'-dependent tests: - Rewrote some to avoid the dependency - Cloned the remainder into new 'ZODB.tests' modules on the ZODB trunk. - - Added support for continuous integration using 'tox' and 'jenkins'. - - Added 'setup.py dev' alias (installs 'nose' and 'coverage'). - - Dropped dependency on 'zope.testing' / 'zope.testrunner': tests now run with 'setup.py test'. Documentation Changes - - - - Refactored many doctests as Sphinx documentation (snippets are exercised via 'tox'). - - Added 'setup.py docs' alias (installs 'Sphinx' and 'repoze.sphinx.autointerface'). I would like to release a '4.0.0' version of the package, and switch the ZODB trunk to pull it in as a dependency (deleting the currently included (older) copy of persistent). One possible issue is that I have not (yet) made the C extensions work under Python 3.2: I don't know whether that should be a blocker for a release. Comments? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/vPyMACgkQ+gerLs4ltQ4DsgCfRsKhBzvTWHehVI86I5sryhjF whQAoL3evyI6GKOfa7eDVjTUVh7B8Q+/ =82vd -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Call for volunteers: help w finishing Python BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/08/2011 06:31 PM, Pedro Ferreira wrote: > >> Really, just getting the tests to pass. I think there are a lot >> of legacy, but still supporte features that need to be fixed. >> (This is a really old package.) > > If it's just a matter of making the tests pass, maybe this will > help? Can you explain the rationale for dropping the use of '_marker' in favor of 'None'? Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAk6gX10ACgkQ+gerLs4ltQ4XGwCfdm2XHVAyjuP1q3E3cfbwPisJ Cr0Al3yRi2sn3nNWLmf52Bv+cnppVng= =+ipL -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] notes re trying to run ZODB on PyPy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/24/2011 08:02 PM, David Glick wrote: > Alan Runyan asked me to post my notes from my attempt to get ZODB > running on PyPy. It's very much an experimental work in progress > (that I got distracted from), but hopefully this is at least useful > to anyone else who wants to attempt the same thing. > > I first tried building ZODB trunk (with C extensions). Ran into the > following issues: * PyPy is missing an implementation of the > _Py_ForgetReference macro which is used by ZODB's > persistent/cPickleCache.c * The check_argument_cmp check in > BTrees/objectkeymacros.h fails for the root object of the database on > PyPy * I got a PyPy exception re wrap_objobjargspec which appeared to > be some issue with using the __setitem__ slot of an extension type > > At this point I gave up on trying to build the C extensions, and > instead did the following: 1. Started with Jim's python-btrees branch > (http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) 2. > Replaced the 'persistent' package with a checkout of Tres' > pure-Python persistent implementation > (http://svn.zope.org/repos/main/persistent/trunk/persistent) 3. Set > exts = [] in setup.py to disable all C extensions when installing. > > Next I worked around some issues in the pure-Python branches: * > BTrees.___BTree.fsBucket class was missing toString and fromString > methods, which I implemented. * persistent.picklecache.PickleCache > class was missing update_object_size_estimation method (and indeed > the whole limit-cache-size-by-bytes feature). I added it as a no-op > stub. * persistent.picklecache.PickleCache.__setitem__ was raising a > KeyError for a duplicate oid even when trying to set the same object > already stored under that key * persistent.TimeStamp did not give the > Python implementation, which I fixed by an import of > persistent.timestamp + module alias if persistent.TimeStamp (the C > module) is missing * persistent.pyPersistence.Persistent's __new__ > needs to accept *args and **kw > > At this point I was able to start up the Pyramid ZODB scaffold and > add an object to the DB root. > > Next I tried to get zodbshootout running as a benchmark. I didn't get > this working, as I got stuck on an issue with the pure-Python > PersistentMapping getting committed without its object state > (appeared to be something to do with failing to get unghosted before > it gets committed). Along the way, I noticed a couple > incompatibilities with PyPy because PyPy implements the cPickle > module as an alias to the pure-Python pickle module. This leads to > some problems due to slight semantic differences between cPickle and > pickle. In particular, cPickle.Pickler accepts a lone protocol > argument; pickle.Pickler does not (leading to an error in > ZEO.zrpc.marshal.encode). And pickle.Unpickler does not have a noload > method, which ZODB.serialize.referencesf uses to obtain the > referenced oids from a pickle without the overhead of unpickling. > > That's as far as I got. Hope this is helpful to the next guy. :) Cool, great work. Are the changes you needed to make available as patchers, or checked in? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5/RgAACgkQ+gerLs4ltQ7NmgCg2n4GT1wQWxPaRNWCguOtAb8q OQ4AoKcbDGep0ig5ZL/x+/m4hNeAVyuq =KtIZ -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Using MySQLProxy with RelStorage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The proxy allows splitting read-write requests, e.g., to allow SELECT queries to run against read-only replicas: http://forge.mysql.com/wiki/MySQL_Proxy Does anyone have experience with using the proxy in conjunction with RelStorage? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5zaTMACgkQ+gerLs4ltQ5sbACePh/J/22LbGFQmdtH/GywlP3b v9sAoNGyos68UHj6krGVf/8fKoo/WyYY =NTrP -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] I would like append an indexer with uuid key at root of my ZODB, how can I append an function to commit event to update this uuid catalog ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/28/2011 05:21 AM, Stéphane Klein wrote: > In repoze.catalog documentation, I read : > > """ > Note that when you call index_doc, you pass in a docid as the first > argument, and the object you want to index as the second argument. When > we index the peach object above we index it with the docid 1. Each docid > must be unique within a catalog; when you query a repoze.catalog > catalog, you’ll get back a sequence of document ids that match the query > you supplied, which you’ll presumably need to map back to the content > object in order to make sense of the response; you’re responsible for > keeping track of which objects map to which document id yourself. > """ > Url : http://docs.repoze.org/catalog/overview.html#indexing > > "you’re responsible for keeping track of which objects map to which > document id yourself." => it is this part I would like to implement. repoze.catalog ships with a 'DocumentMap' which can generate and track the docis for you. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAk4JypkACgkQ+gerLs4ltQ4i4QCXfmxdKLy2nJ3CgdZQeF+Li14F twCdGKTNvpv8A8/dn7ThUAlgjkURkNw= =50dg -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Large ZODB packing and progress
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/25/2011 07:17 AM, Shane Hathaway wrote: > Whereas FileStorage packs by creating a new copy of the database, > RelStorage deletes individual rows. Unfortunately, SQL delete > statements tend to be surprisingly expensive. > > To get closer to what FileStorage does, RelStorage could theoretically > copy rows into new tables instead of deleting rows. I've pondered doing > that, but it would take some major work, I think. Maybe copying would be followed by truncating the original tables and re-populating them via INSERT FROM? Trse. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.co -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1nuhQACgkQ+gerLs4ltQ43PgCghmO5BM2qNznPdBUtuaoIcht8 v+QAn2usSMP84/K25Txl3sILWdtAV18y =UaLr -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate install problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/15/2011 12:06 PM, Tres Seaver wrote: > On 02/15/2011 11:37 AM, Chris Fanning wrote: >> On Tue, Feb 15, 2011 at 5:16 PM, Adam GROSZER wrote: > >>> Hello, >>> >>> You got to use python 2.6, at least for zodbupdate. >>> >>> >> Thanks. I'd managed to get it going with 2.5 but I was getting this >> warning.. > >> Warning: Missing factory for Persistence PersistentMapping > >> Now, using python 2.6, I'm still getting the same warning. >> Looks like I need to get this missing factory because the resulting zodb.fs >> still gives the same error >> AttributeError: 'PersistentMapping' object has no attribute 'has_key' > >> How do I install/configure this? > > You need to make the class importable at the "old" location in order to > use instances of that class to convert to the new location. Something > like:: > > import persistent.mapping > import sys > sys.modules['Persistence.PersistentMapping'] = \ > sys.modules['persistent.mapping'] > > Then you should be able toload you old 'pages' object and create a new > one (using the new class) and copy the contents into it from the old one. Actually, you can install the 'Persistence' package directly (it is distributed separately from Zope): http://pypi.python.org/pypi/Persistence Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1as8MACgkQ+gerLs4ltQ7yzgCeKxXGqx/EIhxfR2BWDtZnOWpF qSoAn3T323/8RIS8wvYYRXmUX5n+lG2J =dRWP -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate install problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/15/2011 11:37 AM, Chris Fanning wrote: > On Tue, Feb 15, 2011 at 5:16 PM, Adam GROSZER wrote: > >> Hello, >> >> You got to use python 2.6, at least for zodbupdate. >> >> > Thanks. I'd managed to get it going with 2.5 but I was getting this > warning.. > > Warning: Missing factory for Persistence PersistentMapping > > Now, using python 2.6, I'm still getting the same warning. > Looks like I need to get this missing factory because the resulting zodb.fs > still gives the same error > AttributeError: 'PersistentMapping' object has no attribute 'has_key' > > How do I install/configure this? You need to make the class importable at the "old" location in order to use instances of that class to convert to the new location. Something like:: import persistent.mapping import sys sys.modules['Persistence.PersistentMapping'] = \ sys.modules['persistent.mapping'] Then you should be able toload you old 'pages' object and create a new one (using the new class) and copy the contents into it from the old one. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1asnoACgkQ+gerLs4ltQ4zkwCgzodbX4J3d4Ig+bmk4Ld09k4j c40AnA9bOnrromUUb4ZI8tgicIUAHHVp =u1n2 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Request for merge: tseaver-squelch_2.6_warnings
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim, While doing work this morning on another branch, I came up with the following small changes which suppress unnecessary warning output when running the tests under Python 2.6: http://svn.zope.org/ZODB/branches/tseaver-squelch_2.6_warnings/ Please review and merge. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1S1IgACgkQ+gerLs4ltQ5rggCfeX5OqhdmvaFF9qkLMLEKm//v HtAAn2tgZ+AT92blJLYBNnKr1ST4zWHw =jTWs -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo server error message
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/31/2011 07:27 AM, Sylvain Viollon wrote: > Hello, > > I have a customer who run a ZEO setup with ZODB 3.8.4. They report > this error: > > 2011-01-31T12:02:24 Unexpected error > Traceback (most recent call last): > File > "/opt/zope/newbuildout/parts/zope2/lib/python/ZODB/ConflictResolution.py", > line 207, in tryToResolveConflict > resolved = resolve(old, committed, newstate) > File > "/opt/zope/newbuildout/parts/zope2/lib/python/ZODB/ConflictResolution.py", > line 141, in __cmp__ > raise ValueError( > ValueError: can't reliably compare against different > PersistentReferences > > Does anyone know to what this error relate ? > > They have been running their ZEO setup for quite a while, and never > experienced any issue. That doesn't seems to 'break' the database either. > > I am not really sure what to say about this error. I believe this is a real bug: the PersistentReference class is raising a ValueError becuase the object being compared is not itself, which implies an irresolvable conflict: it should be raising ConflictError, or else a custom error which would be caught by 'tryToResolveConflict' and converted into a ConflictError. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1G4aYACgkQ+gerLs4ltQ7LMQCdG1u1RF2kWZbhkgwkBbfKTkNy U9wAoMYQ0Tv/SgiXNlCxuUDZaJWjWAnU =PH8G -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Increasing MAX_BUCKET_SIZE for IISet, etc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/27/2011 05:09 AM, Matt Hamilton wrote: >> Conflicts inside the same bucket can be resolved and you won't get to >> see any log message for them. If you get a ConflictError in the logs, >> it's one where the request is being retried. > > Great. That was that I always thought, but just wanted to check. So in > that case, what does it mean if I see a conflict error for an IISet? Can > they not resolve conflicts internally? IIRC: unlike normal adds / removes, bucket splits can cause un-resolvable conflicts (because more than one persistent object's state is involved). In that case, arger buckets should make splits less frequent, and therefore less likely to conflict. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1BnigACgkQ+gerLs4ltQ5nlQCfVxx3V+vtUGWDsyv4iyd/2CRg v0IAn3sfsBbD2iCJ2lwJzU5i5/IK8e+w =xMIT -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Bug (?) in zope/publisher/publish.py:unwrapMethod
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/25/2011 12:19 PM, Vincent Pelletier wrote: > Hi. > > In publish.py[1], unwrapMethod tried to find what can be used to publish an > object. In a site, I had someone create a very-badly-named "func_code" > external method in a place accessible by acquisition from every page on the > site (this bad by itself, and I corrected it already). This caused > unwrapMethod to think it can use any object directly for publishing, because > of: > > elif getattr(unwrapped, 'func_code', None) is not None: > break > > and "unwrapped" is still in an acquisition context. > > Shouldn't the checks be done on unwrapped (from acquisition context) objects > instead, to prevent such stupid mistake to have such a wide impact. > > I have the intuition that this could even be a security problem, allowing an > unexpected object to be called instead of another, but I cannot come out with > an example. > > Do you think there is anything to fix in zope.publisher ? If so, I'll open a > bug. > > [1] > http://svn.zope.org/zope.publisher/trunk/src/zope/publisher/publish.py?view=markup The 'zope.publisher' package doesn't know anything about acquisition-wrapped objects: fixing this issue there would require adding either a dependency on 'Acquisition' (a bad idea, I think) or else adding logic like the 'im_func' bit which peeled away 'aq_self' until all wrappers were gone (using knowleged of Acquisition's API / semantics without importing it). The Zope2-specific version of 'mapply()' (in 'ZPublisher.mapply') is the right place to fix this issue, if it is to be fixed: it could ensure that the acquisition wrapper was stripped away before calling 'unwrapMethod()'. P.S. This issue is off-topic for the ZODB list: I have cross-posted to 'zope-dev': please follow up there. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0/H5sACgkQ+gerLs4ltQ4jdQCgzI+yNAuaX9cqPg77FLHxbzUG abwAn0BJoQ5ImlTrocRMY3KT3M2royOz =WSyR -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Lazy Instantiation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/22/2011 08:54 AM, frits.swinkels wrote: > I am new to ZODB and to this list: greetings. > > I need to get some points clear about precisely when objects are fully > instantiated in memory. It may be a simple matter of pointing me at some > documentation; I read pretty much anything I could find but there may a > problem of your jargon being different from mine:) > > The main data structure of my application is a directed graph of nodes; > each node has at least three slots which contain other nodes (e.g. > parent, right child, left child; the parent always exists, the child > slots may be None). I only want to instantiate that part of the graph > that I am working on. > > Q1. I assume that when an object is instantiated, its "sub" objects will > be in the form of some kind of proxy (perhaps that ghost I met in the > ZODB How document?). Is this assumption correct? Yes. The ghost has not had its instance dictionary populated yet. Attempting to access any instance attribute will "unghostify" the object. > Q2. Given an object, designated by the variable a, in which of the > following statements does the child become instantiated (activated?) ? > child = a.leftchild > name = child.name The second (assuming that the nodes in your data structure derive from a "persistent" base class, such as Persistent, or one of the BTrees), > Q3. Actually, before the child is instantiated, I somehow would like to > query the object a, whether there is anything in the slot, i.e. whether > the content of the slot is not None. Of course, this query should not > trigger the instantiation. That should already be working, again assuming your objects derive from Persistent. > I had the application running in another OO language. An object would > first carry a proxy with the OID in a slot; it would get instantiated on > access. The ad-hoc database made use of the Reiser file system, which > allowed indexing serialized objects on their OID in a B tree. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk065S4ACgkQ+gerLs4ltQ510wCfSujRaHMG7utqgsgRbEEHwmiI iFsAoKEv4K5tIPYR2su/o3aEUDyVfTta =2Jam -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB packing and possible side effects?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/09/2010 05:32 AM, Mark Van den Borre wrote: > I'm trying to read up on ZODB packing and possible side effects. Does > anyone know about comprehensive docs on this? Think interaction with > repozo backup processes, a pack removing weakly referenced objects, > impact on the Data.fs index, ... - - Repozo detects packs, and forces a "full" backup after one, even if you request an incremental. - - You need to explain what you mean by "weakly referenced". - - Pack causes the index to be written. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0A90oACgkQ+gerLs4ltQ6tKwCfdd/XWm5zsYhuEUid7NxCmS1W sLEAoNI6YRioiukHoV7kEtDSGM+ifxsS =u09G -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storing ordered elements
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/02/2010 01:14 PM, Pedro Ferreira wrote: > >> OOSets are ordered. >> > > Yes, I know. My problem was that I wanted to order the elements by > timestamp, and I didn't want to wrap anything around them just to have > them ordered. But I think that in the end that's what I will do... or > just make them comparable by timestamp. One alternative would be to store them as a tuple, (timestamp, object), which should be super-cheap, and still allow for "normal" comparisons of your domain objects. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzQTIoACgkQ+gerLs4ltQ40ywCeKZ34CiZ5889xZ4QeM0nKGNYT nKgAoM8V+9nXjY6aG+3wQrq6GhbXW4bg =UHQx -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo delete_old_backups
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 DJL wrote: > repozo delete_old_backups does not delete old index files Thanks for noticing this issue. I have updated the script and tests to cover this case on the trunk. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkyTbGQACgkQ+gerLs4ltQ5kzQCeIrktIiJc7zaiXjfoNtItyqjX oM8AoL0+Lq7IhNb6KpeD0/6Zhz8KM1IH =at14 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fstest missing as script
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > On Mon, Aug 30, 2010 at 9:45 AM, Adam GROSZER wrote: >> Hello, >> >> Is it intetional that fstest.py is missing from [console_scripts]? >> Anyone against adding it? > > Many of the scripts in ZODB and ZEO were one-offs that should never have been > checked in. Most are not tested and many are not well thought out. > Eventually, I'll find time to go through them all and remove ones that should > be > removed and whip others into shape. > > For this reason, I object to adding fstest as a console script. That objection seems a little odd, given the following checkin message for the file: $ svn log -r 71254 $ZSVN/ZODB/trunk/src/ZODB/scripts/fstest.py r71254 | jim | 2006-11-21 17:01:52 -0500 (Tue, 21 Nov 2006) | 3 lines Moved the scripts down into subpackages to they can be installed via entry points. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkx74+AACgkQ+gerLs4ltQ7x8QCffSVpbFxTJ5cyDRjRO7mydpar NnwAoJl6OCE3IBKtgEkibAIn4TKVgYiE =2u8C -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] weird error while packing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Adam GROSZER wrote: > This exception came up while packing a filestorage: > > 2010-08-29T05:05:23 ERROR ZODB.DB packing > Traceback (most recent call last): > File "D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\DB.py", line > 826, in pack > self.storage.pack(t, self.references) > File > "D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\FileStorage.py", > line > 1126, in pack > pack_result = self.packer(self, referencesf, stop, gc) > File > "D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\FileStorage.py", > line > 1079, in packer > opos = p.pack() > File > "D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\fspack.py", > line 398, > in pack > self.gc.findReachable() > File > "D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\fspack.py", > line 191, > in findReachable > self.findReachableAtPacktime([z64]) > File > "D:\home\.buildout\eggs\ZODB3-3.9.5-py2.5-win32.egg\ZODB\FileStorage\fspack.py", > line 276, > in findReachableAtPacktime > for oid in self.findrefs(pos): > SystemExit > > Any pointers appreciated. Hmm, that looks like the SystemExit got raised as a result of this code:: tlen = self._read_num(pos) if tlen != th.tlen: self.fail(pos, "redundant transaction length does not " "match initial transaction length: %d != %d", tlen, th.tlen) which would suggest some corruption in the filestorage. I would try running the read-only diagnostics on the file to see what they say (e.g., fstest, fsrefs, etc.). How that 'self.fail' got converted to a SystemExit is a mystery. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkx6i9AACgkQ+gerLs4ltQ5VRwCgphZhIjwoCB8zfsDr8XlYrxzS mycAoKlD02s6nloHElBXozy4tEyRNGtZ =FSNN -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] db.undoLog() does not work as documented ..
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > On Mon, Aug 23, 2010 at 12:23 PM, steve wrote: >> Hello Jim, >> >> On 08/23/2010 08:59 PM, Jim Fulton wrote: >>> On Sat, Aug 21, 2010 at 5:39 AM, steve wrote: >>>> Hi, >>>> >>>> I was reading up on the zodb docs, specifically about undoing >>>> tansactions[1] > > It's worth noting that these are not "the" docs. I didn't write or > review them. I don't have any control over zodb.org. I have no idea > how to comment on the docs. (I could possibly find out, but I don't have time > to work that hard.) > >>>> where it mentions that ... >>>> >>>> """ >>>> undoLog(start, end[, func])() method on the DB instance returns the log >>>> of past >>>> transactions, returning transactions between the times start and end, >>>> measured >>>> in seconds from the epoch. >>>> """ >>> That documentation is incorrect. >>> >>> See the definition of undoLog in ZODB/interfaces.py >>> >> Thanks for clarifying that. I've filed a bug: >> >> https://bugs.launchpad.net/zodb/+bug/622828 > > This is problematic. I didn't write the docs and the docs > are not part of the software. I can't do anything about this > I don't know if anyone who can deal with bugs has any control > over zodb.org. I'm pretty sure that site is built via Sphinx from the 'zodbdocs' quasi-package in the Zope SVN: $ svn co $ZSVN/zodbdocs/trunk Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkxyqwMACgkQ+gerLs4ltQ5xeACfaQEwEce9JCyzusB2UkGy9TxP QvoAmwSxvpkYgm00VNl6xxAzJiUSKbZf =jNPl -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: > On Mon, Aug 16, 2010 at 12:14 PM, Pedro Ferreira > wrote: >> Could this be some problem with using persistent objects as keys in a BTree? >> Some comparison problem? > > I'm not entirely sure about this, but I think using persistent objects > as keys isn't supported. Looking at the code, I doubt using anything > expect simple types like unicode strings or tuples of simple types > will work without further work. > >>From what I can see in the code, BTree's use functions like > PyObject_Compare to compare different keys. Persistent doesn't > implement any special compare function and falls back to the standard > hash algorithm for an object. This happens to be its memory address. > The memory address obviously changes over time and the same address > gets reused for different objects. > > I think implementing a stable hash function for your type could make > this work though. > > The ZODB gods correct me please :) Btrees require "comparability", rather than "hashability": your persistent type needs to define a total ordering[1], which typically means defining '__cmp__' for your class. You could also define just '__eq__' and '__lt__', but '__cmp__' is slightly more efficient. [1]http://www.zodb.org/documentation/guide/modules.html#total-ordering-and-persistence Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkxpK18ACgkQ+gerLs4ltQ5gIwCfQDXxLwxpWrghFtRF76MBxesS RtcAnA3+yv+6+XfDRkPBBZcZpoqZ6mO0 =ygVN -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Merge request: ZODB/branches/tseaver-better_repozo_tests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Tres Seaver wrote: >> Jim Fulton wrote: >>> On Fri, May 14, 2010 at 1:57 PM, Tres Seaver wrote: >>> ... >>>> Jim, would you please review and merge the 'tseaver-better_repozo_tests' >>>> branch to the ZODB trunk? The only non-test change is the one I'm >>>> replying to, which is intended purely to make stuff more unit testable. >>> Done >> >>>> Once this merges, I can do the "index saving" work on top of the new tests. >>> Cool. Thanks. >> >> I have landed that feature on a new branch: >> >> http://svn.zope.org/ZODB/branches/tseaver-repozo_index/ >> >> Please review and merge, if ready. >> >> Note that the index restoration feature cannot work in the default case >> (no output file specified, ergo data written to stdout). > > Merged. Thanks! >> Note that the index restoration feature cannot work in the default case >> (no output file specified, ergo data written to stdout). > > I wonder if anyone uses the write-to-standard-output mode. > It seems like an attractive nuisance. It enables some kinds of sysadmin-y solutions, like doing the backup over SSH when you don't have disk space on the host machine. We might make it not be the default, and require that you spell it '-o -' or something. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvxlEsACgkQ+gerLs4ltQ6ffQCeNfst5goiks18P7bGHtPgqJTm FwEAmgOdQnMIZHcPXOZzCnIx3X8cTGV1 =99fp -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Merge request: ZODB/branches/tseaver-better_repozo_tests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > On Fri, May 14, 2010 at 1:57 PM, Tres Seaver wrote: > ... >> Jim, would you please review and merge the 'tseaver-better_repozo_tests' >> branch to the ZODB trunk? The only non-test change is the one I'm >> replying to, which is intended purely to make stuff more unit testable. > > Done > >> Once this merges, I can do the "index saving" work on top of the new tests. > > Cool. Thanks. I have landed that feature on a new branch: http://svn.zope.org/ZODB/branches/tseaver-repozo_index/ Please review and merge, if ready. Note that the index restoration feature cannot work in the default case (no output file specified, ergo data written to stdout). Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvu2fwACgkQ+gerLs4ltQ45/ACeNQm49PSe862DLfECnGgsTP3w b/0An3Gl65Pt7YKE3lJQh9xkM1gFytFq =bdIy -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Merge request: ZODB/branches/tseaver-better_repozo_tests
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Tres Seaver wrote:
> Log message for revision 112305:
> Hooks for testability:
>
> Raise an exception from 'do_full_backup' / 'do_incremental_backup' where
> we would have called sys.exit(2) to avoid overwriting files. Catch the
> exception in 'main' and do the same error reporting / exit.
>
> Allow the 'options' object to pre-set the time used for generating the
> backup filenames.
>
>
> Changed:
> U ZODB/branches/tseaver-better_repozo_tests/src/ZODB/scripts/repozo.py
>
> -=-
> Modified: ZODB/branches/tseaver-better_repozo_tests/src/ZODB/scripts/repozo.py
> ===
> --- ZODB/branches/tseaver-better_repozo_tests/src/ZODB/scripts/repozo.py
> 2010-05-14 17:35:52 UTC (rev 112304)
> +++ ZODB/branches/tseaver-better_repozo_tests/src/ZODB/scripts/repozo.py
> 2010-05-14 17:43:39 UTC (rev 112305)
> @@ -93,6 +93,9 @@
> VERBOSE = False
>
>
> +class WouldOverwriteFiles(Exception):
> +pass
> +
> def usage(code, msg=''):
> outfp = sys.stderr
> if code == 0:
> @@ -301,7 +304,9 @@
> ext = '.deltafs'
> if options.gzip:
> ext += 'z'
> -t = time.gmtime()[:6] + (ext,)
> +# Hook for testing
> +now = getattr(options, 'test_now', time.gmtime()[:6])
> +t = now + (ext,)
> return '%04d-%02d-%02d-%02d-%02d-%02d%s' % t
>
> # Return a list of files needed to reproduce state at time options.date.
> @@ -419,8 +424,7 @@
> options.full = True
> dest = os.path.join(options.repository, gen_filename(options))
> if os.path.exists(dest):
> -print >> sys.stderr, 'Cannot overwrite existing file:', dest
> -sys.exit(2)
> +raise WouldOverwriteFiles('Cannot overwrite existing file: %s' %
> dest)
> log('writing full backup: %s bytes to %s', pos, dest)
> sum = copyfile(options, dest, 0, pos)
> # Write the data file for this full backup
> @@ -447,8 +451,7 @@
> options.full = False
> dest = os.path.join(options.repository, gen_filename(options))
> if os.path.exists(dest):
> -print >> sys.stderr, 'Cannot overwrite existing file:', dest
> -sys.exit(2)
> +raise WouldOverwriteFiles('Cannot overwrite existing file: %s' %
> dest)
> log('writing incremental: %s bytes to %s', pos-reposz, dest)
> sum = copyfile(options, dest, reposz, pos - reposz)
> # The first file in repofiles points to the last full backup. Use this
> to
> @@ -570,7 +573,11 @@
> argv = sys.argv[1:]
> options = parseargs(argv)
> if options.mode == BACKUP:
> -do_backup(options)
> +try:
> +do_backup(options)
> +except WouldOverwriteFiles, e:
> +print >> sys.stderr, str(e)
> +sys.exit(2)
> else:
> assert options.mode == RECOVER
> do_recover(options)
Jim, would you please review and merge the 'tseaver-better_repozo_tests'
branch to the ZODB trunk? The only non-test change is the one I'm
replying to, which is intended purely to make stuff more unit testable.
Once this merges, I can do the "index saving" work on top of the new tests.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkvtjxoACgkQ+gerLs4ltQ7HYgCfU3DBrVQZXQMX0Qgz1YwdLo/T
44cAmwVHWlVvfKGH4miQe4nnxFc82Yd7
=7SLd
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo full backup and index files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tres Seaver wrote: > Hanno Schlichting wrote: >> On Fri, May 14, 2010 at 10:24 AM, Christian Theune wrote: >>> Hmm. If the full backup is just a regular FS file then you could start >>> with the naive approach and just open/close it once after performing a >>> backup as that would create the index file. >> Sure. That would be an easy but also rather inefficient way. In my >> case it takes only about 15 minutes of rather heavy I/O to create the >> file, but for larger files this time usually goes up. > >> >From what I understand repozo does the following: > >> 1. it opens the real file storage file as a read-only FileStorage >> 2. calculates the byte position of end of the last complete transaction >> 3. closes the file storage >> 4. opens the fs file as a normal binary file >> 5. copies over all bytes up to the calculated position into a temp file >> 6. closes the fs file and temp file and renames the temp file >> according to some timestamp > >> What I'm wondering is, if you could copy or otherwise create the index >> as part of the second step. In step 5 we only deal with bytes and >> cannot parse those, but in step 2 we have a proper ZODB FileStorage >> object and can to some degree decide which transaction we want. > > This strategy should be workable for both full and incremental backups: > the storage opened in step 1 has an attribute, '_index', which is an > fsIndex instance. We should be able to save the index into the > repository direcotry with a timestamped filename, something like:: > >index_file = os.path.splitext(dest)[0] + '.index' >fs._index.save(fs._pas, index_file) > > At restore time, we would look for the index corresponding to the full > or incremental being restored, and copy it into place followi[ng the > successful copy of the data. > > As usual, the tedious part about implementing this feature will be > adding / updating tests. I would be willing to make a stab at this, if we can hold off on a 3.10.0 beta until I've had a chance to try it. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvtX0gACgkQ+gerLs4ltQ4dWQCgs3Icg+omT7O3xfxCPoIiIyCm zuEAoKprV5ggwEx9xZHC7bWVX5Zfuogc =AKk0 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo full backup and index files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: > On Fri, May 14, 2010 at 10:24 AM, Christian Theune wrote: >> Hmm. If the full backup is just a regular FS file then you could start >> with the naive approach and just open/close it once after performing a >> backup as that would create the index file. > > Sure. That would be an easy but also rather inefficient way. In my > case it takes only about 15 minutes of rather heavy I/O to create the > file, but for larger files this time usually goes up. > >>From what I understand repozo does the following: > > 1. it opens the real file storage file as a read-only FileStorage > 2. calculates the byte position of end of the last complete transaction > 3. closes the file storage > 4. opens the fs file as a normal binary file > 5. copies over all bytes up to the calculated position into a temp file > 6. closes the fs file and temp file and renames the temp file > according to some timestamp > > What I'm wondering is, if you could copy or otherwise create the index > as part of the second step. In step 5 we only deal with bytes and > cannot parse those, but in step 2 we have a proper ZODB FileStorage > object and can to some degree decide which transaction we want. This strategy should be workable for both full and incremental backups: the storage opened in step 1 has an attribute, '_index', which is an fsIndex instance. We should be able to save the index into the repository direcotry with a timestamped filename, something like:: index_file = os.path.splitext(dest)[0] + '.index' fs._index.save(fs._pas, index_file) At restore time, we would look for the index corresponding to the full or incremental being restored, and copy it into place followi[ng the successful copy of the data. As usual, the tedious part about implementing this feature will be adding / updating tests. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvtV0EACgkQ+gerLs4ltQ4JFgCfYZ3PV4pZCP7NInG1lW36F0ke sRMAoL24vUmOIpb2W2XOps7I9TuxmGRk =aMi4 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction 1.1.0 released
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > I've made a new transaction release with the changes we've discussed > recently, as well a threading-related bug fix. > > The retry iterator goodness won't work with ZODB until the next ZODB > release (after 3.10.0a2). Great! BTW, I had asked the owner of the 'transaction' project on Launchpad to transfer ownership to the 'zope-admin' group. He did so, and I have configured the project to track its own bugs, as well as updating the metadata, etc. I made the 'zodb-developers' group the "driver" for the project. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvsHjQACgkQ+gerLs4ltQ68FwCfZtesBqPZUZDpZwSiSxM1don5 FfgAoIuy3mTM7P2tfVrwMIXJ/H2bnmbB =gbdp -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jim Fulton wrote:
> On Mon, May 10, 2010 at 3:27 PM, Ryan Noon wrote:
>> Here's my code:
>>
>> self.storage = FileStorage(self.dbfile, pack_keep_old=False)
>> cache_size = 512 * 1024 * 1024
>>
>> self.db = DB(self.storage, pool_size=1, cache_size_bytes=cache_size,
>> historical_cache_size_bytes=cache_size, database_name=self.name)
>> self.connection = self.db.open()
>> self.root = self.connection.root()
>>
>> and the actual insertions...
>> set_default = wordid_to_docset.root.setdefault #i can be kinda
>> pathological with loop operations
>> array_append = array.append
>> for docid, wordset in docid_to_wordset.iteritems(): #one of my
>> older sqlite oodb's, not maintaining a cache...just iterating (small
>> constant mem usage)
>> for wordid in wordset:
>> docset = set_default(wordid, array('L'))
Note that you are creating the array willy-nilly in the inner loop here.
I would nearly always write that as::
docset = wordid_to_docset.root.get(wordid)
if docset is None:
docset = array('L')
wordid_to_docset.root[worid] = docet
>> array_append(docset, docid)
Why are you using an unbound method here? The following would be
clearer, and almost certainly not noticeably slower:
docset.append(docid)
>> n_docs_traversed += 1
>> if n_docs_traversed % 1000 == 1:
>> status_tick()
>> if n_docs_traversed % 25000 == 1:
>> self.do_commit() #just commits the oodb by calling
>> transaction.commit()
Don't forget the final commit. ;) Also, I don't know what the 'array'
type is here, but if it doesn't manage its own persistence, then you
have a bug here: mutating a non-persistent sub-object doesn't
automatically cause the persistent container to register as dirty with
the transaction, which means you may lose changes after the object is
evicted from the RAM cache, or at shutdown.
> Also note that memory allocated by Python is generally not returned to
> the OS when freed.
Python's own internal heap management has gotten noticeably better about
returning reclaimed chunks to the OS in 2.6.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkvoeIoACgkQ+gerLs4ltQ6WjACgsvDmG96nD2iPl/noiHS5ThdL
SdIAn1Ei+yfzRyJ4W1lwvuThBj9BxzGt
=nrBB
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Plone-Users] question about using repozo and blob storage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ralf Hemmecke wrote: > Tres Seaver palladion.com> writes: > >> Sharing the blobs using a DVCS pull sounds like an interesting approach. >> One flaw with your outlined implementation is that ZODB blob files are >> *never* updated: rather, if a ZODB object updates the blob object, it >> creates a new blob file and stores its ID. Old blob files are cleaned >> up during a pack. > > Interesting. You know what git does? If one adds a file X (actually git deals > with contents not files) to the repository, git computes a hash value (sha1) > and > uses that value as the name for the file. It's stored under the .git > directory. > Some update of bookkeeping data and that's it. > If the contents of the file X changes, and it is committed to the git > repository, git computes a sha1 and stores the new content under that new sha1 > name + updating of the bookkeeping. The old file is not deleted. Git does not > even bother to compress anything. That would be done by an additional "git > gc". > In contrast to most other SCMs git can be considered as a database. That git > is > mostly recognized as a version control system, is only half of the story. The > underlying storage system is what make git so interesting. > > I have no good knowledge of zodb and you are certainly doing great with it, > but > my concern is with the blobs (which as I hopefully understand correctly are to > be stored outside of Plone's Data.fs). > Storing versions of files into a DVCS is like freeing the zodb (for blobs) > from > doing its own versioning. > > Actually, I think conceptually git and the zodb are quite similar. One > transaction is nothing else than a commit in the SCM sense. > >> If you want to experiment with the idea anyway, then writing a "wrapper" >> storage is probably the way to go: DemoStorage is one such beast, as is >> the BlobStorage itself. > > I'm sorry, I cannot do this. I've simply no time for opening up another area > of > development for me. I understand that completely: I'm there myself. ;) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvodS4ACgkQ+gerLs4ltQ68JACgnReABZhAcIOcpxADMftidN6D WK4AniKQUv7LU1X1fSqFY8gFZ3xuhllA =J5Q0 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Plone-Users] question about using repozo and blob storage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > On Mon, May 10, 2010 at 2:21 PM, Tres Seaver wrote: > ... >> If you want to experiment with the idea anyway, then writing a "wrapper" >> storage is probably the way to go: DemoStorage is one such beast, as is >> the BlobStorage itself. > > I don't consider the wrapper approach to have been very effective for > BlobStorage, although it was a reasonable thing to try at the time. > BlobStorage is likely to go way. > > (This comment should not be construed as supporting the idea of implementing > some sort of blob-storage back end using DVCS.) I'm having troube with the last sentence: do you really mean you don't like the idea of blobs-in-DVCS? Or did you drop the second part of a double negative inadvertently? Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEUEARECAAYFAkvodNQACgkQ+gerLs4ltQ6qYQCfSUDolO7AgmwnpsE75oYSAYVT 4mAAmMGLaO/HvEEwsGLtxYV4U8vevBk= =2VUg -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Plone-Users] question about using repozo and blob storage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ralf Hemmecke wrote: > Subject: Re: [Plone-Users] question about using repozo and blob storage > Date: Fri, 7 May 2010 10:05:59 -0700 (PDT) > From: ajung > To: plone-us...@lists.sourceforge.net > > > This list is the wrong place for such deep ZODB related discussions. Suggest > this on the zodb-dev list. > > -aj > > > Ralf Hemmecke-2 wrote: >> On 05/07/2010 06:40 PM, ajung wrote: >>> Blobs are - as David explained - a _core_ component of the ZODB. Content >>> stored through the file storage or blob storage are handled and treated >>> in >>> the same way to the application. If you want to write a SCM-based backend >>> for ZODBI am not sure how far you will get and if it would be >>> reasonably >>> fast and without major architectural issues. >> I understand that meta data lives still in ZODB. And I wouldn't want to >> suggest that people deal with the blobs in any other way than through >> plone. One should only be able to have read access to the blobstorage >> (if it's not through plone, otherwise disaster may happen). >> >> I haven't actually looked into how blobs are actually stored on the file >> system, i.e., in particular what name they will have and whether the >> file system structure is the same as the folder hierarchy inside plone. >> But that doesn't actually matter. >> >> I assume that they are stored in some directory in a directory for >> blobs, let's call it blobdir. >> >> I speak now for git, but other SCMs are quite similar (maybe not svn). >> >> This blobdir is turned into a repository by simply saying "git init". >> >> Then when plone stores a file, it will write it into (a subdir of) >> blobdir. With GIT behind that should read: >> >> (1) write it into blobdir and then >> (2) git add FILENAME" >> (3) git commit -m "some message" >> >> That's it. >> >> If the user uploads a new file to plone, then plone doesn't have to care >> about renaming the old file and storing the new, it simply has to do THE >> SAME steps (1)..(3) as above. I call that easier. >> >> Maybe packing is a bit more difficult and depends on the SCM in >> question. For git it would be a simple "git filter-branch" (of course >> with appropriate parameters) followed by "git gc". >> >> Just my two cents. I would find such a GIT backend wonderful. But I've >> no experience with how that blob stuff is actually done through >> plone/zope. So sorry if I cannot provide any code. Sharing the blobs using a DVCS pull sounds like an interesting approach. One flaw with your outlined implementation is that ZODB blob files are *never* updated: rather, if a ZODB object updates the blob object, it creates a new blob file and stores its ID. Old blob files are cleaned up during a pack. If you want to experiment with the idea anyway, then writing a "wrapper" storage is probably the way to go: DemoStorage is one such beast, as is the BlobStorage itself. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvoTrgACgkQ+gerLs4ltQ5HBgCgwXNqzE4KYWAQZYEJMPSoi18t 09EAniD4ucOThPmJRJ8A9TjmVuQv3gQV =QUON -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] mkzeoinstance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > ZODB includes a script named mkzeoinstance that tries to create > a somewhat self-contained ZEO server installation. > > It only works when ZODB and its dependencies are installed in > in the standard Python path (typically site-packages). > > It only works on unix-like systems. > > It has no tests. > > This is an example of an unsupported feature. It probably needs to be > heavily documented as such. If someone wants a project, it would be > nice to give mkzeoinstance the love it needs. > > If no one is going to support this properly, I'd rather just drop it. > I do think mkzeoinstance has some potential value, so I'd prefer to > see someone maintain it. I don't think it would take *that* much > effort to whip it into shape, but I have too many other commitments to > take this on myself. I can offer some suggestions is someone picks it > up. > > Another option would be to move it to a separate project that depends > on ZEO/ZODB. At least then, I wouldn't be responsible for not > maintaining it. :) > > Thoughts? Volunteers? Forked to: http://svn.zope.org/zope.mkzeoinstance and released: http://pypi.python.org/pypi/zope.mkzeoinstance/3.9.4 Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvQmGYACgkQ+gerLs4ltQ5HvQCfds4/2GXeffhWOuZ9zsSod4PU loMAoKMIlsztuppgBP3Ch8Zk+xzgLUmA =SZ4M -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Request for merge: tseaver-lp143158-feature branch
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > On Mon, Apr 12, 2010 at 6:29 AM, Tres Seaver wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Implements '--kill-old-on-full' command line switch for repozo, which >> removes older full / incremental backups on successful completion of a >> full backup. >> >> http://svn.zope.org/ZODB/branches/tseaver-lp143158-feature/?rev=106808&view=rev > > Done. > > Jim > > P.S. Do I owe you any other ZODB merges? I just pushed up a branch fixing the 'mkzeoinst' script, which was broken by the move of the 'zdaemon' package out into its own egg. http://svn.zope.org/ZODB/branches/tseaver-lp_143361/ The branch also adds support (from the LP #143361 bug) for supplying a host interface in addition to a port to the script. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvKFxcACgkQ+gerLs4ltQ6qXACeKBvv6gZ9YbGZQjwNRomvrdLo Pq8An0nLoZ6U56dcXbsOM41LxFOLzROU =Ih5U -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Low-level reference finding, tracking, deleting?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff Shell wrote: > We encountered a problem during an export/import in a Zope 3 based > application that resulted in something not being importable. This is > from our very first Zope 3 based application, and I stumbled across > some very old adapter/utility registrations that I thought I had > cleared out. There are references to `zope.interface.adapter.Null` > which haven't been around for years. This is in an old > `LocalAdapterRegistry` which, again, I thought I had removed along > time ago. These objects and what they reference are not part of our > normal object graph, and I was surprised to see them. > > Given an oid, how can I trace what references that object/oid? There > is something in our normal object hierarchy retaining a reference, > but I don't know how to find it, and imagine that trying to > investigate/load the objects from the ZODB level will help me find > the culprit. > > Are there low level deletion tools in the ZODB to delete individual > objects? Hmm, maybe you could hack a dummy version of 'zope.inteface.adapter.Null' back into place, and set a breakpoint inside its __setstate__? Just a thought. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvJbEUACgkQ+gerLs4ltQ4kRQCgkfoqc/n3HNVetNr0Iz68I5mn ciwAoMAkXTLIUiFLjI8q85z4/mLYOrSA =U5u1 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nitro wrote: > Hello Tres, > > thanks for your detailed answers! > > Am 12.04.2010, 22:42 Uhr, schrieb Tres Seaver : > >>> Additionally I made some quick performance tests. I committed 1kb sized >>> objects and I can do about 40 transaction/s if one object is changed per >>> transaction. For 100kb objects it's also around 40 transactions/s. Only >>> for object sizes bigger than that the raw I/O throughput seems to start >>> to >>> matter. >> 40 tps sounds low: are you pushing blob content over the wire somehow? > > No, that test was with a plain file storage. Just a plain Persistent > object with a differently sized string and an integer attribute. I did > something like > > 1) create object with attribute x (integer) and y (variably sized string) > 2) for i in range(100): obj.x = i; transaction.commit() > 3) Measure time taken for step 2 > >>> Still don't know the answers to these: >>> >>> - Does it make sense to use ZODB in this scenario? My data is not suited >>> well for an RDBMS. >> YMMV. I still default to using ZODB for anything at all, unless the >> problem smells very strongly relational. > > Ok, the problem at hand certainly doesn't smell relational. It is more > about storing lots of different data than querying it extensively. It's a > mixture of digital asset management (the blobs are useful for this part) > and "projects" which reference the assets. The projects are shared between > the clients and will consist of a big tree with Persistent objects hooked > up to it. I have seen the ZEO storage committing transactions at least an order of magnitude faster than that (e.g., when processing incoming newswire feeds). I would guess that there could have been some other latencies involved in your setup (e.g., that 0-100ms lag you mention below). >>> - Are there more complications to blobs other than a slightly different >>> backup procedure? >> You need to think about how the blob data is shared between ZEO clients >> (your appserver) and the ZEO storage server: opinions vary here, but I >> would prefer to have the blobs living in a writable shared filesystem, >> in order to avoid the necessity of fetching their data over ZEO on the >> individual clients which were not the one "pushing" the blob into the >> database. > > The zeo server and clients will be in different physical locations, so I'd > probably have to employ some shared filesystem which can deal with that. > Speaking of locations of server and clients, is it a problem - as in zeo > will perform very badly under these circumstances as it was not designed > for this - if they are not in the same location (typical latency 0-100ms)? That depends on the mix of reads and writes in your application. I have personnally witnessed a case where the clients stayed up and serving pages over a whole weekend in a clusterfsck where both the ZEO server and the monitoring infrastructure went belly up. This was for a large corporate intranet, in case that helps: the problem surfaced mid-morning on Monday when the employee in charge of updating the lunch menu for the week couldn't save the changes. >>> - Are there any performance penalties by using very large invalidation >>> queues (i.e. 300,000 objects) to reduce client cache verification time? >> At a minimum, RAM occupied by that queue might be better used elsewhere. >> I just don't use persistent caches, and tend to reboot appservers in >> rotation after the ZEO storage has been down for any significant period >> (almost never happens). > > In my case the clients might be down for a couple of days (typically 1 or > 2 days) and they should not spend 30 mins in cache verification time each > time they reconnect. So if these 300k objects take up 1k each, then they > occupy 300 MB of ram which I am fine with. If the client is disconnected for any period of time, it is far more likely that just dumping the cache and starting over fresh will be a win. The 'invalidation_queue' is primarily to support clients which remain up while the storage server is down or unreachable. >>> From what I've read it only seems to consume memory. >> Note that the ZEO storage server makes copies of that queue to avoid >> race conditions. > > Ok, I can see how copying and storing 300k objects is slow and can take up > excessive amounts of memory. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Exc
Re: [ZODB-Dev] Using zodb and blobs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nitro wrote: > After lots of googling and browsing the source I can answer some of my > questions: > >> - What's the difference between storing bigger objects as blobs and as >> plain large strings? > > Plain large strings cannot be streamed for instance. Products like Zope > chop up their file uploads into 64kb chunks which are then stored as > individual objects in the zodb. That was the strategy before blobs. ZODB versions since 3.8 support storage of BLOb data as files on the filesystem. >> - Can I stream in parts of a blob/large string without having to read all >> of it? > > I can get a file handle to a blob. Strings are always read as a whole. > >> - Where can I find example code on zodb blobs? E.g. how do I save a blob, >> how do I read it back in? > > The ZODB/tests directory features a few blob doctests which provide all > the necessary code to get started. Having this on zodb.org would be nice > (especially since the doctests are already ReST-formatted). > > Additionally I made some quick performance tests. I committed 1kb sized > objects and I can do about 40 transaction/s if one object is changed per > transaction. For 100kb objects it's also around 40 transactions/s. Only > for object sizes bigger than that the raw I/O throughput seems to start to > matter. 40 tps sounds low: are you pushing blob content over the wire somehow? > Still don't know the answers to these: > > - Does it make sense to use ZODB in this scenario? My data is not suited > well for an RDBMS. YMMV. I still default to using ZODB for anything at all, unless the problem smells very strongly relational. > - Are there more complications to blobs other than a slightly different > backup procedure? You need to think about how the blob data is shared between ZEO clients (your appserver) and the ZEO storage server: opinions vary here, but I would prefer to have the blobs living in a writable shared filesystem, in order to avoid the necessity of fetching their data over ZEO on the individual clients which were not the one "pushing" the blob into the database. > - Is it ok to use cross-database references? Or is this better avoided at > all cost? I would normally avoid them out of habit. They seem to work, though. > And new questions: > > - Does the _p_invalidate hooking as outlined at > http://www.mail-archive.com/zodb-dev@zope.org/msg00637.html work reliably? Never tried it, nor felt the need. > - Are there any performance penalties by using very large invalidation > queues (i.e. 300,000 objects) to reduce client cache verification time? At a minimum, RAM occupied by that queue might be better used elsewhere. I just don't use persistent caches, and tend to reboot appservers in rotation after the ZEO storage has been down for any significant period (almost never happens). > From what I've read it only seems to consume memory. Note that the ZEO storage server makes copies of that queue to avoid race conditions. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvDd5sACgkQ+gerLs4ltQ5h4wCghqTQsNO/5XrqHMZyhi8Hz17d oRcAn1el86604KoTTWB8Bx5R13ZlvQB/ =momg -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Testing gocept.zeoraid
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Sylvain Viollon wrote: >I am currently testing gocept.zeoraid, and this seems to work fine. > >However, I have few questions: > >- On the pypi the last released version is 1.0b7. In the Zope SVN it > seems to be only 1.0b1. Hmm, I see tags for 1.0a1, b1-b7:: $ svn ls $ZSVN/gocept.zeoraid/tags 1.0a1/ 1.0b1/ 1.0b2/ 1.0b3/ 1.0b4/ 1.0b5/ 1.0b6/ 1.0b7/ and the trunk changelog shows 1.0b8 as unreleased:: $ svn cat $ZSVN/gocept.zeoraid/trunk/CHANGES.txt | head Change History == 1.0b8 (unreleased) -- - #484437: Recipe handles address in zeo.conf incorrectly. When giving\ the ZEO address on the server only as a port that port would be ignored\ resulting in 127.0.0.1:8100 to be assumed for the management scripts. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkungIwACgkQ+gerLs4ltQ4bSwCfXumBlBylXUIpt1kJC7s85StG o6wAn1IyqebgEroNngKJ5jg06A7qx0XS =CwE5 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persistance class problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 rohit wrote: > I keep getting an error : > "TypeError: issubclass() arg 1 must be a class" > > On debugging i found that its due to the issubclass method being passed a > nontype object instead of a class. > During the debugging process i find that the above mentioned error is due to > the persistance class.The exact message i get is : > "Unable to get repr for " > > I posted this error last time and was told to provide more info. > Hope this helps someone crack this. Your previous traceback was: > "D:\python\lib\site-packages\zodb3-3.10.0a1-py2.6-win32.egg\ZODB\Connection.py", > line 245, in get > obj = self._reader.getGhost(p) > File > "D:\python\lib\site-packages\zodb3-3.10.0a1-py2.6-win32.egg\ZODB\serialize.py", > line 576, in getGhost > if issubclass(klass, Broken): > TypeError: issubclass() arg 1 must be a class which indicates that you have an invalid pickle: the connection is trying to create the "ghost" for an object (an empty instance) before loading its state, but the part of the pickle for the object's class is not actually a class. What type of object was the "nontype object" you mentioned above? Another of things to look at: Are you loading the data using the exact same versions of code used to store the data? If not, then look for places in the code where a class might have been replaced in the code with a factory function (or other callable) of the same name. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkujLzoACgkQ+gerLs4ltQ4/RgCdFtJ0YHFP1xT2h3L099L49x3M HVUAoMyWY6+duWcr13SdwYoHbj1rmfGT =j/2+ -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Merge request: repozo cleaning up old files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim, I would like to see this branch merged into a release of ZODB: $ZSVN/ZODB/branches/tseaver-lp143158-feature/ The branch fixes the tests for repozo to make them true unit tests, as well as adding a long-missing feature (optionally cleaning out old backup sets when making a new full backup). Ideally, I would prefer that it go into a new 3.9.x release, since 3.9.x is the version most people are still using in production. If the risk seems too high (I think any risk here is very negligible), then I would at least like to see it in a 3.10.x release. Thanks! Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuZYOoACgkQ+gerLs4ltQ7qoQCgl69xbHzZkvzs0mGcGF3ilAGo rqcAnArLyAevGzu7zNtYCQEABjpa76/w =JL48 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Get rid of zope.proxy dependency
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: > We used zope.proxy to implement blob storages as storage proxies. It > seemed like a good idea at the time, but since then I've added a > mix-in class that I've used in a number of storage implementations. > > Do we still need the proxy-based implementation class? If we do, then > I plan to at least switch to using a simple pure-python proxying > mechanism. If no one speaks up in favor of keeping the proxying > support, I'll probably just drop it. +1 to dropping it. Tres. - -- ======= Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuQFKAACgkQ+gerLs4ltQ7jdgCfZfW70Ihn2vIC2lEECFnMmy// KjsAnRcx6T0J+SPuj+P1MZcOa1tlghH5 =hjYY -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Location of transient caches
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andy Theyers wrote: > Hi all > > We have a Debian server running 3 Zope instances, each mounting its > database from a single ZEO instance on the same server. > > Until very recently we had been running each Zope with a persistent > cache (using the 'client' directive in the zeo_client stanza of > zope.conf). Each cache was 300MB, as recommended by various buildout > scripts and other "how to" documents I had read. > > While under this configuration the site failed due to cache corruption. > This caused us to remove the persistent cache option, again as > recommended in various mailing list posts. > > However, as we are using the default Debian disk partitioning (whereby / > is only 1GB) we ran out of disk space, as the transient caches appear to > be being created in /tmp > > Is there any way I can change the location of the transient cache files, > putting them on a partition with more space? I've been scouring the > source but am yet to come up with the bit that creates the temporary > cache files. It's not an area of Zope that I've got any experience of. > > Any help would be greatly appreciated. The cache uses the 'tmepfile' module, which uses a number of environment variables to select the location: http://docs.python.org/library/tempfile.html Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt+tUEACgkQ+gerLs4ltQ4uHgCeOIV05O6lwvsUdeiOnUpgekoc Zi0AoJvZ1zCbAO0ehx25uQX/krgvJ7X0 =IyXE -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Multi-Threaded Access
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jared Smith wrote: > I'm looking at using ZODB and have two different databases stored in > different files where different threads will be periodically > insert/update/delete various records in both databases. Couple of questions > to make sure this is feasible: > > When one of the threads wants to do an insert of a set of records I need to > ensure that those changes are submitted in a transaction that does not > overlap with the records being inserted in another thread. Looks like the > default API assumes there is a single global transaction > (transaction.commit()) however I saw mention of creating your own > transaction.manager that you can bind at DB.open() time. > > Does this mean to support this model I have to re-open the database and do > the binding of a transaction manager in every method where I want to do this > transactional action? For the moment I was assuming a DB access object that > is shared (singleton access class). > > Is the better model to have something like a pool of DB access classes where > each one maintains a DB handle and a transaction and one of the access > classes would be taken out of the pool and used every time a transaction > needs to be run? Worried a bit about the overhead of re-opening the DB is > this not something to worry about? This second pattern is the one commonly used in Zope and other ZODB-based web apps: each request gets a "connetion" object from a pool maintained by the database, and uses the objects in its cache to satisfy the request. If the request completes successfully, the application then commits the transaction, other wise it aborts. In either case it returns the connection to the pool at the end of the request. The current transaction for a request is stored in a thread local. Committing a transaction can fail (raise a ConflictError) if another, overlapping transaction has modified some of the same objects: in this case, the application thread has the option of synchronizing its connection's cache and retrying the request (Zope tries up to three times). Hope that helps, Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktiemEACgkQ+gerLs4ltQ4jVgCeLUC3nyLPToguuFujL8fe06Jg 8T0AoIR8ShyGrHDiq41DsmGBKgLs+KxV =Rm/D -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persistent object has empty __dict__ for a little while
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Martin Aspeli wrote:
> Hi,
>
> This one is pretty high no the list of weirdest things to have happened
> to me in a while. Basically, I have a persistent object that has an
> empty __dict__() on the first request, until it suddenly decides to have
> data again.
>
> I'm on ZODB 3.9.3, using Zope 2.12 and Plone 4. I have a persistent
> object, /plone/portal_registry:
>
>class Registry(registry.Registry, SimpleItem):
>pass
>
> The base class is:
>
>class Registry(Persistent):
>def __init__(self):
>self._records = Records(self)
>
> Records is:
>
> class Records(Persistent):
>
>__parent__ = None
>
>def __init__(self, parent):
>self.__parent__ = parent
>self.data = OOBTree()
>
>def __getitem__(self, name):
>return self.data.__getitem__(name)
>
> The BTree contains string keys and Record objects as values. Record is:
>
> class Record(Persistent):
> __name__ = u""
> __parent__ = None
>
> field = None
>
> __parent__ is set to records.__parent__, which is going to be the
> Registry object.
>
> field is set to a special persistent field class inheriting from
> zope.schema's fields, but mixing in persistence. In this case, I have a
> list field:
>
>class PersistentField(persistent.Persistent):
>"""Base class for persistent field definitions.
>"""
>
>class PersistentCollectionField(PersistentField,
> zope.schema._field.AbstractCollection):
>
>class List(PersistentCollectionField, zope.schema.List):
>pass
>
> I then have some code like this:
>
> def cloneField(self, field):
> clone = field.__class__.__new__(field.__class__)
> clone.__dict__.update(field.__dict__)
> for name, attr in field.__dict__.items():
> if IField.providedBy(attr):
> clone.__dict__[name] = self.cloneField(attr)
> return clone
>
> This is used like so:
>
> registry = getUtility(IRegistry) # persistent/local
> clone = self.cloneField(registry.records[recordName].field)
>
> This seems to work always *except* on the first request immediately
> after starting up Zope. In this case, field.__dict__ is {}. If I poke at
> it long enough in pdb, it suddenly springs into life and contains values
> again.
>
> Can anyone enlighten me as to:
>
> - why this may happen
> - how I can ensure it stops happening :-)
You have a ghost, likely: try getting it to unghostify by accessing an
attribute, or calling '_p_activate' (accessing '__dict__' doesn't
provoke the auto-activation).
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAktE7UUACgkQ+gerLs4ltQ68YwCeOywUw4a/c7UXap6uxyIzMCx7
9VcAoII4X56V7LConhNSYCWvFMWMooBD
=XvFD
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees and id mapping (was Re: Data.fs size grows non-stop)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Pedro Ferreira wrote: > David Glick wrote: >> Pedro Ferreira wrote: >> >>> Hello all, >>> >>> So, I've been experimenting with a new structure for our date indexes, >>> and I was considering using an IOBTree that would map integer timestamps >>> to BTree-based sets. As for the sets, I considered two options: >>> >>> >> Be careful with using timestamps as IOBTree keys. By default this type >> of BTree uses the system integer, which on many systems is 32 bits. >> That means that you'll overflow the maximum allowed key for timestamps >> beyond the year 2038. It won't even give an error; it will just >> silently overflow and overwrite existing values in the BTree. >> > Yes, I know. That's one of the reasons I'm using mmdd integers > instead (the other one being i only need 1 day resolution). > > Thanks a lot for the tip! An LOBTree uses a guaranteed-to-be-long (64 bits) integer for the keys, if you need that. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksmsuMACgkQ+gerLs4ltQ6mNgCgwBpVRaUli7a7Ij1NQYYHsxif OD0AnjgAGY1TB6GAadx1qSmwW/8w5c1a =ZJRe -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Godefroid Chapelle wrote: > Godefroid Chapelle wrote: >> Jim Fulton wrote: >> >> >> >>> No one seems to be willing to step up and take responsibility. >>> I guess everyone assumes I will. >>> >>> Jim >>> >> I'll take the responsibility of transforming the manual test to an >> automated... >> > > Done in svn.zope.org/repos/main/ZODB/branches/test_repozo > > I added code changes from revisions 105888, 105915 and 105919. > > Tests pass for me on MacOSX with 2.4, 2.5 and 2.6 Thanks for throwing yourself on that grenade, Godefroid. I have checked in changes to your branch which: - - Use a tempdir for backup / restore. - - Clean out module-scope imports. - - Avoid shellling out to run repozo, but rather use its main(), passing argv. - - Suppress noisy output, unless NOISY_REPOZO_TEST_OUTPUT is in the environment. Remaining cleanup would be to remove the "Monte Carlo" use of random to generate permutations: the test takes a *long* time to run, because it tries to run 100 times and use random.random to get permutations of the three choices run: these should be six separate tests, likely. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksWuYsACgkQ+gerLs4ltQ7LYwCg0AK9vHca71u7q0maPM86hY2l yzkAoIzgiGiEZLhcuG7vvTnHtTeuDbK2 =QKtA -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris Withers wrote: > Martin Aspeli wrote: >> - there were no tests for the existing code > > There are tests, they're just manual and spew a lot of output even when > passing, but they do (as best I could tell) exercise the backup and > restore cycle quite heavily. Jim is vetoing any changes to repozo until soembody updates that test to run as an automated unit test. I find this a strange exercise in leadership: this tool, which is important to much of the ZODB community, happens to be one Jim *doesn't use at all* (because of ZRS). A sufficiently cynical observer might even squint at that and see a potential conflict of interest. > I ran these before and after to verify they didn't fail but did exhibit > the behaviour I was trying to fix before I started and didn't fail *and* > didn't exhibit the behaviour I was trying to fix after. > >> - Tres wrote a trivial test for the trivial fix > > Indeed. His test exhibited the problem prior to the fix and showed it to > be fixed after. (since the problem was a warning generated on import) > >> - You now found some different problem in the same code, > > ...but didn't say what it is. Jim seemed to hint it was a Python 3 > problem. I'd warrant there are much more serious problems to be worrying > about for Python 3 first. If Python 3 compatibility is now a requirement > for any patches submitted, then we're well and truly fucked. Sniffing at sys.versioninfo was a silly way to do the fix anyhow (I think Jim even suggested it): much better would have been to conditionally import hashlib, and fall back to md6 only where it raises an import error. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksWkp8ACgkQ+gerLs4ltQ46AgCguhm0exzl8YKj3I7nt2/IRU99 OxEAoK88bwOputbisPASL3Mdt2dzPtM7 =815V -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jim Fulton wrote:
> On Fri, Nov 20, 2009 at 11:49 AM, Tres Seaver wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Tres Seaver wrote:
>>> Jim Fulton wrote:
>>>> On Fri, Nov 20, 2009 at 5:49 AM, Chris Withers
>>>> wrote:
>>>>> Hi Jim,
>>>>>
>>>>> It would be great if the two recent fixes I committed could make it into a
>>>>> 3.9.4 release some time soon...
>>>>>
>>>>> What can I do to help this happen?
>>>>> Assuming I do anything necessary, when could you do this release?
>>>>> (or give Christian permission to do it :-) )
>>>> I need to review the changes before the release. I'll probably reject the
>>>> repozo change without an automated test.
>>> -sys.maxint. In what possible universe are we better off not being able
>>> to run repozo under Python 2.6?
>
> You can run repozo under Python 2.6 now. You'll get a deprecation warning.
So we have an annoyance rather than a catastrophe: that helps some. It
will still break on 2.7 without the trivial fix.
>>> That fix is trivial to inspect for
>>> correctness, changes no behavior, and is like dozens of other Python 2.6
>>> compatiblitity changes made in our codebase.
>>>
>>> Repozo is *the* documented way of backing up filestorages:
>
> Where is that?
http://wiki.zope.org/ZODB/FileStorageBackup
>>> with Zope
>>> 2.12 and later now running on Python 2.6, we should just tell people
>>> "Sorry, you can't do backups"?
>
> No, we should tell them they should use a mostly untested tool to do backups
> that doesn't work with blobs. That should inspire confidence.
We can use rsync to back up blobs safely / losslessly without restarting
the ZEO server: we can't do that with filestorages.
> Does Zope 2.12 use Blobs?
If configured, sure.
>> I just checked in a stupid test to ensure that repozo can be imported.
>>
>> - --- src/ZODB/scripts/tests.py (revision 105913)
>> +++ src/ZODB/scripts/tests.py (working copy)
>> @@ -26,6 +26,11 @@
>> (re.compile('hash=[0-9a-f]{40}'),
>> 'hash=b16422d09fabdb45d4e4325e4b42d7d6f021d3c3')])
>>
>> +class RepozoTests(unittest.TestCase):
>> +
>> +def test_importability(self):
>> +from ZODB.scripts import repozo
>> +
>
> I appreciate your dedication to quality.
>
> That test passes under ZODB 3.9.3 with Python 2.6.
It emits the same DeprecationWarning (the "bug" Withers fixed). I did
have to munge the 'test_suite' dinosaur as well.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAksG4bIACgkQ+gerLs4ltQ7czQCgzAjQX0TUEJKSNo92+uv8UWOw
VmEAoJ9Xo7WRamCQyuwmY7Kacrqk/Krz
=/5ad
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
