Re: [ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/04/2014 06:28 AM, Godefroid Chapelle wrote: Le 03/02/14 20:53, Tres Seaver a écrit : I wish you hadn't pushed that -- some of these changes are definitely inappropriate on the 3.10 branch (adding an Acquisition dependency is definitely wrong). Tres. Acquisition is added as a test dependency. Any hint how to replicate the bug without acquisition is welcome. Define a subclass of Persistent which emulates what Acquisition does, e.g.: from persistent import Persistent class Foo(Persistent): @property def _p_jar(self): # or whatever attribute trggers return object() Which other change is inappropriate ? Adding MANIFEST.in on a release branch seems wrong to me (I don't like them anyway, and we *definitely* don't want to encourage instsall-from-a-github-generated-tarball on a release branch). Overall, these changes would have been better pushed on a separate branch, which could then be merged into the appropriate release branches *and* the trunk. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLxJukACgkQ+gerLs4ltQ4iEwCfSMrYBnSJbm9JzTJoqeb84lWk v2YAnRAqkPFJ5zcVscm8C4qymwSjeeQI =OEB0 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/03/2014 08:30 AM, GitHub wrote: Author: Godefroid Chapelle got...@bubblenet.be Author: Godefroid Chapelle got...@bubblenet.be Author: Godefroid Chapelle got...@bubblenet.be Author: Godefroid Chapelle got...@bubblenet.be Author: Godefroid Chapelle got...@bubblenet.be Author: Godefroid Chapelle got...@bubblenet.be I wish you hadn't pushed that -- some of these changes are definitely inappropriate on the 3.10 branch (adding an Acquisition dependency is definitely wrong). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLv87QACgkQ+gerLs4ltQ6XJQCgim/cILFy2rM0qpBO8/wUU0CP TDEAn0rnHd/KqejwVyeuTLiHesrmUDAR =E6G2 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB.FileStorage.format: TxnHeader cannot handle Unicode 'descr'
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/05/2013 01:13 PM, Jim Fulton wrote: On Sat, Oct 5, 2013 at 1:47 AM, Chao Meng bobom...@gmail.com wrote: Hi there, Here is an issue from zopefoundation/ZODB github three months ago. I am facing similar issue now and don't know how to resolve. github issue link: https://github.com/zopefoundation/ZODB/issues/12 When I use Pyramid+ZODB+traversal and use some Chinese characters in URL. Note that my resource tree saved in ZODB with unicode fine for the Chinese characters as object names. Basically when save transaction, ZODB.FileStorage.format TxnHeader uses request.path_info as it's descr, which is unicode, but TxnHeader cannot handle Unicode :( It would be great if anyone can help or give some pointers. This is a Pyramid bug. transaction.note is defined to take a bytes string. Pyramid should encode the path before passing it to transaction.note. The interfaces says text. I realize that this is likely for hysterical raisins, but if we mean bytes, we should say so. Note that the implementation's use of an unadorned string literal to join the values means that in Py3k, it really *is* text, and not bytes. If we want the application to do the encoding, then we should change that literal as well. Alternatively, Pyramid could store the path in transaction extended info, which accepts any picklable type. Of course, we could revisit this. If we did, I'd deprecate the transaction user and description attributes and only support meta data via the extended info mechanism, which I'd rename meta data. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJS2g0ACgkQ+gerLs4ltQ5WEwCfdRPMCIrR5GdVaiPgLiBJT7QT 9wIAoLQzqhB9HRWYYIjsAx77+9A0dngc =PFfZ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZODB 4.0.0b3] DB.pack() broken?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 08/14/2013 02:05 PM, Tres Seaver wrote:
On 08/14/2013 07:48 AM, [mabe] wrote:
Hi everyone,
I noticed today that database packing doesn't seem to work (at least
on Python 3.3)
Minimal example:
from ZODB import FileStorage, DB
storage = FileStorage.FileStorage('/tmp/mystorage.fs') db =
DB(storage) db.pack() db.close()
Expected: No output / db.pack() should succeed, although there is
likely nothing to pack.
Actual output: % python zodb_pack.py packing Traceback (most recent
call last): File /usr/lib/python3.3/site-packages/ZODB/DB.py,
line 810, in pack self.storage.pack(t, self.references) File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1121, in pack pack_result = self.packer(self, referencesf,
stop, gc) File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1074, in packer opos = p.pack() File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
408, in pack self.gc.findReachable() File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
191, in findReachable self.findReachableAtPacktime([z64]) File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
276, in findReachableAtPacktime for oid in self.findrefs(pos): File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
329, in findrefs return self.referencesf(self._file.read(dh.plen))
File /usr/lib/python3.3/site-packages/ZODB/serialize.py, line 646,
in referencesf u.noload() AttributeError: 'Unpickler' object has
no attribute 'noload' Traceback (most recent call last): File
zodb_pack.py, line 5, in module db.pack() File
/usr/lib/python3.3/site-packages/ZODB/DB.py, line 810, in pack
self.storage.pack(t, self.references) File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1121, in pack pack_result = self.packer(self, referencesf,
stop, gc) File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1074, in packer opos = p.pack() File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
408, in pack self.gc.findReachable() File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
191, in findReachable self.findReachableAtPacktime([z64]) File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
276, in findReachableAtPacktime for oid in self.findrefs(pos): File
/usr/lib/python3.3/site-packages/ZODB/FileStorage/fspack.py, line
329, in findrefs return self.referencesf(self._file.read(dh.plen))
File /usr/lib/python3.3/site-packages/ZODB/serialize.py, line 646,
in referencesf u.noload() AttributeError: 'Unpickler' object has
no attribute 'noload'
Thanks very much for the report. I can reproduce with your example
Christian Tismer submitted a pull request fixing the bug -- I verified,
merged it, and released zodbpickle 0.5.2 with the fix:
https://pypi.python.org/pypi/zodbpickle/0.5.2
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlIP87AACgkQ+gerLs4ltQ4wEACeIvdOrpmAGXvypMC4BVfl/80A
Fc4AniGouCvRK1wlV3Ds/Adx+B/yXo3Z
=OmFf
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbpickle: need a new tag
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/16/2013 03:33 PM, Christian Tismer wrote: Hi Stephan, thanks a lot that you merged my work! Now there is a little bit missing: For some reason, the 0.5.1 tag was not merget from the clone. All files there, but the tag is still 0.5.0. I would like to push it on PyPI, but for that I need a new tag. Can you please add a new tag, like 0.5.1 ? Thanks cheers -- chris Guys, the 0.5.1 releas / trunk has test failures under Py3k: https://github.com/zopefoundation/zodbpickle/issues/6 In addition to the problem just reported (Unpickler instance has no attribute 'noload'). In both cases, the 0.5.0 tag / release does not have the problem. Can you have a look? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlILyPEACgkQ+gerLs4ltQ4akQCbBTFaNf4JnLzFZ3m3s12DfxrO TJYAnA5mDGyfbpeSL+jzFjJnMF3R50dJ =Vd3B -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Attempts at Python2-compatible pickles cause unpicklable objects
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/08/2013 02:03 AM, Tres Seaver wrote: One point to consider is that anybody who has been using ZODB on Py3k since the beta will have to do surgeery on their filestorages to get them to load after that change (the code doesn't make any attempt to determine if the magic number is later; it just checks for an exact match). I don't want to make the wait, we have five users, we can't take out the tab requirement mistake here: anybody for whom this would be a hardship needs to speak up pretty quickly. I merged the 'issue_10' branch to trunk and released ZODB 4.0.0b3 yesterday. Today I found (as I expected) that I couldn't open the database I created last week under Py3k after updating. Am I the only person affected? Or do we need to add fallback code to permit opening a filestorage with an older magic number in read-only mode? In my case, the data is all test stuff, so I can just blow it away and recreate it. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlG4rpcACgkQ+gerLs4ltQ4h/gCcCc/PBE6rY0FcJ2+pXqAAXpEJ SNkAn1v3O3ypcIfQYN+egYnBixNZIUii =zQ3t -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Attempts at Python2-compatible pickles cause unpicklable objects
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/07/2013 09:56 AM, Jim Fulton wrote: On Fri, Jun 7, 2013 at 3:02 AM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Jun 06, 2013 at 09:43:16PM -0400, Stephan Richter wrote: On Thursday, June 06, 2013 09:28:38 PM Tres Seaver wrote: If I am correct about this consensus, then in addition to merging PR #11 for ZODB, we should also drop the 'bytes_as_strings' changes to 'zodbpickle'. Yes Yes, this is a left-over from our experimentation. It did not work, since it also pickled attribute names as bytes and when you loaded them, the attributes would be gone. The specific combination last used in ZODB4 on Python 3 (pickle bytes_as_strings=True, load with encoding='ascii', errors='bytes') handled attributes fine. Apparently it broke datetime objects -- and I never noticed. Also, requiring protocol 3 for Python 3 sounds correct as well. How about also bumping the magic number of Data.fs files so ZODB on Python 2 would reject it early? I have made that change on the 'issue_10' branch. One point to consider is that anybody who has been using ZODB on Py3k since the beta will have to do surgeery on their filestorages to get them to load after that change (the code doesn't make any attempt to determine if the magic number is later; it just checks for an exact match). I don't want to make the wait, we have five users, we can't take out the tab requirement mistake here: anybody for whom this would be a hardship needs to speak up pretty quickly. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGyyT8ACgkQ+gerLs4ltQ4SxACgrskSD/t/9R0yFtlsSUQRGIuX 0REAoMWBctMgYYqgu0WiV2E5WCi790o4 =mazo -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RFC: Attempts at Python2-compatible pickles cause unpicklable objects
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I posted an issue, including a gist showing the problem: https://github.com/zopefoundation/ZODB/issues/10 The 'issue_10' branch adds a test (failing without the patch) and a patch for this issue: https://github.com/zopefoundation/ZODB/pull/11 I think we arrived at consensus that trying to emit backward-compatible pickles (bytes pickled using the string opcodes, as in Python2) was the wrong approach, and that when we get around to tackling the problem, we would do so by having Python2 emit forward-compatible pickles instead (using a custom 'binary' type for things which should be bytes in Py3k, and using the protocol 3 opcodes to pickle them). If I am correct about this consensus, then in addition to merging PR #11 for ZODB, we should also drop the 'bytes_as_strings' changes to 'zodbpickle'. Thoughts? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGxN0YACgkQ+gerLs4ltQ7+SgCeNLwvgq56afQnu06kw8bUO30E 69wAnR2WcFwz43N7QmxxBU93zxrcD/aO =/9eb -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 05/25/2013 02:50 AM, Marius Gedminas wrote:
On Fri, May 24, 2013 at 09:31:55PM -0400, Tres Seaver wrote:
On 05/24/2013 01:28 PM, Marius Gedminas wrote:
I was trying to debug a failing ZEO test case
(testConcurrentUpdates), and found out it had nothing to do with
ZEO. There's a bug either in BTrees or in ZODB itself that
causes data corruption under Python 3.x.
Here's a test case:
https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py
Thanks for the detective work! I can confirm the bug, and that it
is in the BTrees C extensions (the tests pass after 'rm
.tox/py32/lib/python3.2/site-packages/BTrees/*.so').
It's the COMPARE macro in _compat.h. This patch fixes the data
corruption:
diff --git a/BTrees/_compat.h b/BTrees/_compat.h index
e004d54..4765df1 100644 --- a/BTrees/_compat.h +++ b/BTrees/_compat.h
@@ -28,8 +28,8 @@ #define TEXT_FORMAT PyUnicode_Format
#define COMPARE(lhs, rhs) \ -PyObject_RichCompareBool((lhs),
(rhs), Py_LT) 0 ? -1 : \ -(PyObject_RichCompareBool((lhs),
(rhs), Py_EQ) 0 ? 0 : 1) +(PyObject_RichCompareBool((lhs),
(rhs), Py_LT) 0 ? -1 : \ + (PyObject_RichCompareBool((lhs),
(rhs), Py_EQ) 0 ? 0 : 1))
#else
I'm now trying to write a failing unit test. It's a bit difficult to
get access to C macros from Python tests ;-)
COMPARE is used in two places:
BTrees/objectkeymacros.h-#define TEST_KEY_SET_OR(V, KEY, TARGET) \
BTrees/objectkeymacros.h:if ( ( (V) = COMPARE((KEY),(TARGET)) ),
PyErr_Occurred() )
and
BTrees/objectvaluemacros.h:#define TEST_VALUE(VALUE, TARGET)
COMPARE((VALUE),(TARGET))
We can ignore TEST_KEY_SET_OR, as it works fine there.
TEST_VALUE is used to implement BTree.byValue and bucket merging.
The bug I was chasing down was caused by bucket mismerges, I think.
It should be easier to write a test for byValue. In fact there is
one, in test__base.py, but it's not run against real C-implemented
BTrees. I copied it over into test_OOBTree.py and discovered that it
fails unexpectedly:
diff --git a/BTrees/tests/test_OOBTree.py
b/BTrees/tests/test_OOBTree.py index bc046bc..e813f88 100644 ---
a/BTrees/tests/test_OOBTree.py +++ b/BTrees/tests/test_OOBTree.py @@
-99,9 +99,15 @@ class OOSetPyTest(ExtendedSetTests,
unittest.TestCase):
class OOBTreeTest(BTreeTests, unittest.TestCase):
-def _makeOne(self): +def _makeOne(self, *args): from
BTrees.OOBTree import OOBTree -return OOBTree() +
return OOBTree(*args) + +def test_byValue(self): +ITEMS =
[(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')] +
tree = self._makeOne(ITEMS) +
self.assertEqual(list(tree.byValue(22)), +
[(y, x) for x, y in ITEMS[22:]])
def testRejectDefaultComparison(self): # Check that passing int keys w
default comparison fails. @@ -146,9 +152,9 @@ class
OOBTreePyTest(OOBTreeTest): # default 'object' comparison semantics.
#class OOBTreePyTest(BTreeTests, unittest.TestCase):
-def _makeOne(self): +def _makeOne(self, *args): from
BTrees.OOBTree import OOBTreePy -return OOBTreePy() +
return OOBTreePy(*args)
The failure is
AssertionError: Lists differ: [(25, 'z'), (24, 'y'), (23, 'x... !=
[(22, 'w'), (23, 'x'), (24, 'y...
First differing element 0: (25, 'z') (22, 'w')
- [(25, 'z'), (24, 'y'), (23, 'x'), (22, 'w')] + [(22, 'w'), (23,
'x'), (24, 'y'), (25, 'z')]
so the C-implemented BTrees return values in descending order, while
the base Python implementation returns them in ascending order. This
happens on Python 2.7 as well, so it's not related to the COMPARE
macro bug.
Does anybody know what the API of BTree.byValue() is supposed to be?
The interface just says Return a sequence of (value, key) pairs,
sorted by value, without clarifying if it means an ascending or a
descending sort.
The docstring in the C module says:
byValue(min) - list of value, key pairs
Returns list of value, key pairs where the value is = min. The
list is sorted by value. Note that items() returns keys in the
opposite order.
So I guess the descending order is intentional: the call to '.reversed'
is certainly there. Let's update the Python version to match, and use
your test to verify both.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlGg3YAACgkQ+gerLs4ltQ5eeACgywzDPvD33nkM4Skd05PSOLMR
WPcAn03amWB8zcsmU+eQKnZBZ6AbYr/d
=7AlH
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/25/2013 11:49 AM, Tres Seaver wrote: On 05/25/2013 02:50 AM, Marius Gedminas wrote: On Fri, May 24, 2013 at 09:31:55PM -0400, Tres Seaver wrote: On 05/24/2013 01:28 PM, Marius Gedminas wrote: I was trying to debug a failing ZEO test case (testConcurrentUpdates), and found out it had nothing to do with ZEO. There's a bug either in BTrees or in ZODB itself that causes data corruption under Python 3.x. Here's a test case: https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py Thanks for the detective work! I can confirm the bug, and that it is in the BTrees C extensions (the tests pass after 'rm .tox/py32/lib/python3.2/site-packages/BTrees/*.so'). It's the COMPARE macro in _compat.h. This patch fixes the data corruption: diff --git a/BTrees/_compat.h b/BTrees/_compat.h index e004d54..4765df1 100644 --- a/BTrees/_compat.h +++ b/BTrees/_compat.h @@ -28,8 +28,8 @@ #define TEXT_FORMAT PyUnicode_Format #define COMPARE(lhs, rhs) \ -PyObject_RichCompareBool((lhs), (rhs), Py_LT) 0 ? -1 : \ -(PyObject_RichCompareBool((lhs), (rhs), Py_EQ) 0 ? 0 : 1) +(PyObject_RichCompareBool((lhs), (rhs), Py_LT) 0 ? -1 : \ + (PyObject_RichCompareBool((lhs), (rhs), Py_EQ) 0 ? 0 : 1)) #else I'm now trying to write a failing unit test. It's a bit difficult to get access to C macros from Python tests ;-) COMPARE is used in two places: BTrees/objectkeymacros.h-#define TEST_KEY_SET_OR(V, KEY, TARGET) \ BTrees/objectkeymacros.h:if ( ( (V) = COMPARE((KEY),(TARGET)) ), PyErr_Occurred() ) and BTrees/objectvaluemacros.h:#define TEST_VALUE(VALUE, TARGET) COMPARE((VALUE),(TARGET)) We can ignore TEST_KEY_SET_OR, as it works fine there. TEST_VALUE is used to implement BTree.byValue and bucket merging. The bug I was chasing down was caused by bucket mismerges, I think. It should be easier to write a test for byValue. In fact there is one, in test__base.py, but it's not run against real C-implemented BTrees. I copied it over into test_OOBTree.py and discovered that it fails unexpectedly: After adding your new test, and wrapping the 'COMPARE()' macro inside extra parens within 'TEST_VALUE' (in 'objectvaluemacros.h'), the new 'byValue' test passes on Py3k. I am trying to reproduce the ZEO test failure now, and then against the BTrees trunk, before making a BTrees release. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGg9rIACgkQ+gerLs4ltQ6M5ACeIyIgCC9/jWwew1qSFyJNg+eb EDIAnRORMJ2XN7xm8kjd+lZciEaJKkA0 =iXx3 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/25/2013 01:36 PM, Tres Seaver wrote: I am trying to reproduce the ZEO test failure now, and then against the BTrees trunk, before making a BTrees release. Rather than relying on the slow and somewhat non-deterministic ZEO failures, I have verified that the current tip of BTrees master permits Marius' 'zodbfail' tests to pass. I will make a BTrees 4.0.7 release shortly. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGg+FcACgkQ+gerLs4ltQ553wCggrDP4WCyWbxcd1LpEqts8fnZ gNgAnRwB/0oB8/psYdSVlXkFAV3GPv8P =q+w9 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/25/2013 01:43 PM, Tres Seaver wrote: I will make a BTrees 4.0.7 release shortly. That would be 4.0.8: https://pypi.python.org/pypi/BTrees/4.0.8 Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGg+dsACgkQ+gerLs4ltQ6jTACgsMxUAD/oFW/h3MtcDw4wYWCU +NIAoJZOBLH4zGZwAbMBUjaSRZuOBJNb =NclA -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 05/24/2013 01:28 PM, Marius Gedminas wrote:
I was trying to debug a failing ZEO test case
(testConcurrentUpdates), and found out it had nothing to do with ZEO.
There's a bug either in BTrees or in ZODB itself that causes data
corruption under Python 3.x.
Here's a test case:
https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py
It initializes an OOBTree to {0:0, 1:0, 2:0}, then launches two
threads that attempt just one write each:
tree[1] = 1 # thread 1 tree[2] = 2 # thread 2
All transactions are committed successfully and the end result
_sometimes_ is {0:0, 1:0, 2:2} instead of the expected {0:0, 1:1,
2:2}.
There's also a larger test case (zodbfail.py) that modifies more than
one item in a thread. That one fails pretty reliably.
This only happens under Python 3.x.
To reproduce::
git clone git://gist.github.com/5644876.git zodbfail cd zodbfail
detox
Thanks for the detective work! I can confirm the bug, and that it is in
the BTrees C extensions (the tests pass after 'rm
.tox/py32/lib/python3.2/site-packages/BTrees/*.so').
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlGgFIsACgkQ+gerLs4ltQ5QxgCgves8VGndIxt37UCV/CPMKeuw
t3EAoK+gwFE3SNOpBSLJ0p+mZ4ftP7gl
=wxp6
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/08/2013 12:34 PM, Tres Seaver wrote: On 04/29/2013 08:37 PM, Stephan Richter wrote: Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready for Py3 with noload() support. I totally agree that we do not need to solve any of the transition work now. So for ZODB Py3 support we need to: 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just contain the cPickle code that is Py3 compatible. I do not care whether this happens for ZODB 4.0 or 4.1 as long as I get some commitment that 4.1 Chris and I chatted with Jim about this over beers last Friday. I explained that the current 'py3; branch does not require the 'zodbpickle everywhere' stuff (the Python2 side doesn't use 'zodbpickle'). Jim then agreed that we could merge that branch before releasing 4.0. We will need to add some caveats to the docs / changelog (Python3 support is only for new applications, no forward- / backward-compatibility for data, etc.) Given that ZODB won't import or use 'zodbpickle' under Python2, I don't think we need to remove the current Python2 support (as released in 0.4.1): the Python3 version (with noload()) has been there all along. I have merged the 'py3' branch to 'master': - - All tests pass under all four platforms using buildout. - - All unit tests pass on all four platforms using 'setup.py test'. I added the following note to the changelog: ZODB 4.0.x is supported on Python 3.x for *new* applications only. Due to changes in the standard library's pickle support, the Python3 support does **not** provide forward- or backward-compatibility at the data level with Python2. A future version of ZODB may add such support. Applications which need migrate data from Python2 to Python3 should plan to script this migration using separte databases, e.g. via a dump-and-reload approach, or by providing explicit fix-ups of the pickled values as transactions are copied between storages. I pushed out a ZODB 4.0.0b1 release after the merge. If the buildbots stay green over the weekend, I think we can release a 4.0.0 final early next week. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGNYN8ACgkQ+gerLs4ltQ5GPACaA3EtwgZOUsvWmsoGOTWi0bpw n5kAoNNUVfsofX/eKxFZKd3F/pLQvvDG =7oxn -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 05/10/2013 07:55 PM, Stephan Richter wrote: On Friday, May 10, 2013 05:04:31 PM Tres Seaver wrote: I pushed out a ZODB 4.0.0b1 release after the merge. If the buildbots stay green over the weekend, I think we can release a 4.0.0 final early next week. Awesome thanks. After the 4.0.0 final release, I will set aside some time for the CH devs to release final versions of all the ZTK packages. Tres, are we ready to commit to a zope.security 4.0.0 as well? AFAIK, we could cut it from the 'master' at any time. I don't know of any issues I don't see any open launchpad issues which should block a release. tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGNtIUACgkQ+gerLs4ltQ6mkACffp0He4ybnf0dkTJwLCiJxtjy b7EAn0S6iQX6sgroQ66VFfFMXeyQOZob =eY19 -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 08:37 PM, Stephan Richter wrote: Well, that's the py3 branch. As Tres mentioned, zodbpickle is ready for Py3 with noload() support. I totally agree that we do not need to solve any of the transition work now. So for ZODB Py3 support we need to: 1. Merge the py3 branch into trunk. 2. Simplify zodbpickle to just contain the cPickle code that is Py3 compatible. I do not care whether this happens for ZODB 4.0 or 4.1 as long as I get some commitment that 4.1 Chris and I chatted with Jim about this over beers last Friday. I explained that the current 'py3; branch does not require the 'zodbpickle everywhere' stuff (the Python2 side doesn't use 'zodbpickle'). Jim then agreed that we could merge that branch before releasing 4.0. We will need to add some caveats to the docs / changelog (Python3 support is only for new applications, no forward- / backward-compatibility for data, etc.) Given that ZODB won't import or use 'zodbpickle' under Python2, I don't think we need to remove the current Python2 support (as released in 0.4.1): the Python3 version (with noload()) has been there all along. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlGKfngACgkQ+gerLs4ltQ500gCfQK4HSzemxaYkcPAyleNdkagq MAwAn3wYoCo4BItBHAve4o+lhrzRTBrt =BbCU -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 09:48 AM, Jim Fulton wrote: On Sun, Apr 28, 2013 at 8:34 PM, Stephan Richter stephan.rich...@gmail.com wrote: On Sunday, April 28, 2013 07:23:12 PM Jim Fulton wrote: Can ZODB 4 be used now without zodbpickle? No, unfortunately for Py2 we need the custom cPickle and for Py3 `noload()` support (as Tres mentioned). This is a problem. The only change in ZODB 4.0 was supposed to be the breakup. This was supposed to be a low-risk release. The separation into multiple packages was supposed to increase agility, but now it appears we're stuck. The only reason we had delayed the 4.0 release (in my mind, anyway) was that it was a good way to signal the Py3k compatibliity changes. I'm not wedded to calling the Py3k-compatible release 4.0. I'd like there to a stable 4.0 release **soon** that doesn't use zodbpickle for Python 2. For now, I suggest we focus on stability and the ability to make progress on non-Python-3-related work. After that is achieved, I suggest we get to the point where people can create new databases and use them with Python 3. We need to do this without hindering the ability to make new stable releases. The trunk of the 'ZODB' package does not have any of the Py3k / zodbpickle changes yet. We could make a ZODB 4.0b1 release from the trunk today and create a '4.0' stable branch prior to any merge of the 'py3' work. As far as the grander vision for Python2/3 transition and interoperability, we need to make progress incrementally and not sacrifice stability of the master branch. I made the 3.11 release fully expecting a stable 4.0 release soon. That was of the 'ZODB3' meta-package, right? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+gboACgkQ+gerLs4ltQ4+7wCg222VrN5b0jkRrSJKVBL1VEBr 5lgAoINrzLbTus6ycBXcVGovxWIPBQ5t =XLcz -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 10:51 AM, Jim Fulton wrote: Right. As I suggested, let's get to a point where we can get a stable ZODB 4.0 release for Python 2. As soon as we get that, let's get a ZODB 4.0.x or 4.1 release that works on Python 3, presumably via zodbpickle. As I proposed earlier this morning, we can make a non-Py3k, non-zodbpickle 4.0b1 release today from the master branch, and a 4.0 final in a week. Once we get that release out, we can then merge the 'py3' branch, including adding the requirement for 'zodbpickle' under both Python2 and Py3k, and aim for a much expedited 4.1 release which supports Py3k. At this point, I would favor putting the database conversion / straddling code into a new package (not 'zodbpickle'). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+iZ4ACgkQ+gerLs4ltQ7SgACfW0UMVtp1MJGplWDEBfx3CM9V pesAoNqece3ZQ/EVoFad6G2+B3vXkCz8 =ig9W -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 11:00 AM, Jim Fulton wrote: Let's keep master stable. Maybe someone will want to add features before the Python 3 support is stable. I don't want to hold 4.1 hostage either. Given that the only folks (besides maybe you) invested in ZODB development want Py3k support ASAP. I don't see that. Do you have features in mind that you would imagine releasing before we land Py3k support? I suggest breaking the Python 3 work into increments that can each be introduced without sacrificing stability. The first increment could provide Python 3 support without any conversion or compatibility support. This is something you could probably achieve pretty quickly and would allow you meet your immediate goals. We are already there, AFAIK, on the 'py3' branch: the blocker is just getting out a release. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+nugACgkQ+gerLs4ltQ4lmACgmfAMS0tqV86+v0ItIlbMkhtK i6gAoLPO4aehpHIrZokPG8c4hRIAOsnE =6f/j -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/29/2013 12:44 PM, Jim Fulton wrote: Yes. There are lots of features I'd like to add to ZODB. I tend to work on them when I have time (infrequently) or where we have a driving need at ZC. Long ZODB release cycles provide a lot of stop energy. We are already developing this way (the 'py3' branch has not been merged). However, if you do a lot of Python2-only feature work and merge to master, you will likely push back to horizon for merging that branch: we will have to port any work done to it. Using the 4.0 label to signal big changes ahead, evaluate carefully before upgrading was the primary reason I had been pushing to get the Py3k stuff landed (the low-risk thing would have been more naturally labeled 3.11). FWIW, we have reports that some brave souls have successfully built Py3k apps using the 'py3' branch. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF+qrEACgkQ+gerLs4ltQ5OpgCg1FZ8xB9EijUQOQqzhm2RAZ4V +z4An29kZZYpoFlFeu0QMpLbqGRfIscV =5EKv -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Which version is the current version of ZODB?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/26/2013 11:51 AM, Matthew Kasfeldt wrote: Hi I am new to the community and not sure where else to ask this. I am learning ZODB and see that there is a ZODB3 and ZODB4. I am unsure of which one is the current stable release of the product or if they are both current but used for different purposes. I on the ZODB site it lists ZODB3 and on wikipedia it states the most current stable is 3.10.3. But looking at various repositories (GitHub and pyppi) I see version 4 is also available (and a 3.11.0a3 alpha release as well). I currently use Python 2.7x and want to be sure I am using the most current stable code to learn on. Any input on the matter would be greatly appreciated! The 3.10.x line is the current stable release line: you want to be installing 'ZODB3==3.10.5'. ZODB 4.x is not yet released: we are hoping to finish the Py3k support soon and make a beta. The code is actually still quite mature, with very good test coverage: it should be stable enough for ordinary development, although I wouldn't want to put it into production just yet. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF6uKwACgkQ+gerLs4ltQ52HgCfU3i5BzzYuFzep0Zl9ZzSh4G7 QQIAn3sxTZIcCe+ZtmQRi7Q0pCq6soJA =F6rH -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 04:38 PM, Tres Seaver wrote: - ``zodbpickle`` should provide a new ``binary`` type which Python2 applications can begin using to signal that attributes should be unpickled in Py3k as ``bytes``. See: https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes - ``zodbpickle`` should provide a pickler/unpickler for use by Python2 clients who operate against converted storages (replace_py2_cpickle_). See: https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes - ``zodbpickle`` should provide a pickler/unpickler for use by Py3k clients who operate against unconverted storages (replace_py3k_pickle_). See: https://github.com/zopefoundation/zodbpickle I have pushed a new branch, 'merge_py2_py3k', which basically holds the existing Py3k forks (from master) alongside the Pyton-2 forks (from 'py2_explicit_bytes'): https://github.com/zopefoundation/zodbpickle/tree/merge_py2_py3k After thrashing and failing in my attempt to have the Python code straddle, I instead have separate versions (based on Python 2.7 and 3.2), and a unifying 'zodbpickle.pickle' module which pulls in the standard API from the appropriate module. This solution is ugly, but it does allow for running tests cleanly on each supported version of Python. It also minimized the extent of patching for each fork (the straddled version I discarded was a franken-module by comparison). I would like to merge this branch to master early next week and make a release, so that we can evaluate merging the 'py3' branch of ZODB. Thoughts? Note that I have not yet addressed the portions of my proposal which deal with analyzing / converting existing databases, or with the possibly-needed wrapper storage (for on-the-fly conversion). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlF68tcACgkQ+gerLs4ltQ4lzQCdHvsnCZ5cfZlycJ1BQw8Uct5q CGYAnilMitegoqsVOevpPlbLcuu7bur6 =qEsj -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 05:13 PM, Stephan Richter wrote: On Tuesday, April 16, 2013 04:38:06 PM Tres Seaver wrote: Comments? (I don't now why Stephan's e-mail didn't make it to the list). The big omission that I noticed while reading the text carefully is a note saying that you will never be able to use stock Py3k pickle, because it does not support noload(). Thus ``zodbpickle`` is needed for any Py3k code. (I think this is a correction to you your last bullet in _replace_py2_cPickle.) Hmm, I think you are correct. That reminds me, originally we forked pickle.py from Python 3.3. During PyCon I think you decided to start by using cPickle from Python 2.7 instead. If you are starting from Py2.7 cPickle, then supporting Protocol 3 is not easy. Already done (as you note in your follow-up). Given your writeup, I think you are implicitly saying to start from Py3.3 pickle and add the special support for Python 2 binary via the special new type. That sounds good to me. I would actually prefer to fork the Python 3.2 version: the one from 3.3 pulls in a bunch of grotty internal-only usage. BTW, what are your motivations for all the different strategies? I wanted to document them all, because some of the strategies suit different cases better than others. _ignore_compat is obvious. If you can easily create the ZODB from other data sources, then you can do a one-time switch. In fact, at CipherHealth we have this case, since the ZODB only contains config (which is loaded from text files) and session data. Yup. Even for large CMS systems, I would still make dump-to-filesystem, then reload a requirement. Others disagree, of course (and may have legitimate reasons). Leo Rochael Almeida has clients with databases too big to convert, for instance (the downtime required to do the conversion would be prohibitive, I believe). But which strategy would be useful for a large Plone site for example? I think we should focus on that and provide one good way to do it. Plone has historically preferred in-place migration to dump-reload. Maybe jumping the Py3k curb is enough reason for them to reconsider. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFu79sACgkQ+gerLs4ltQ4aTQCfSeb6Xiz0OOtoGJuzSKfOetMu 7IAAoMiNzaohY2lv8DMoLRmoNIw8f3P0 =fcYz -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Python2 - Py3k database compatibility
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/16/2013 05:19 PM, Stephan Richter wrote: Oh, I see that in your branch of ``zodbpickle`` you already added protocol 3 support to the Py2.7 cPickle code. Does this code also run under Py3.3? (I don't now why Stephan's e-mail didn't make it to the list). I would not expect to use the Python 2.7-derived pickler on Py3k. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlFu7+UACgkQ+gerLs4ltQ4EWgCgyGHjPlqWEK5+PFhJXK94jCLY edMAn2aZV5dLxV7XdTgSh83yOBrdnTL/ =/tPT -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RFC: Python2 - Py3k database compatibility
on Python2 (Python2 has no support for ``protocol 3``). - - Easiest usage for applications which are never going to straddle. - - Compatibility will only be achievalble via one-time conversions (where the conversion script uses one of the other strategies or tools). .. _replace_py3k_pickle: Replace Py3k ``pickle`` +++ Keep pickling in the Python2 / protocol 1 way we have always done. - - No changes to the ``ZODB`` packages on Python2. Storages do not need to be configured with any custom pickle support. - - On Py3k, ``ZODB`` uses pickler / unpickler from the ``zodbpickle`` module, such that Python2 ``str`` objects are unpickled as ``bytes``; ``bytes`` are pickled using the ``protocol 1`` opcodes (so that Python2 will unpickle them as ``str``). .. _replace_py2_cPickle: Replace Python2 ``cPickle`` +++ Move to pickling in the new protocol 3 way (native under Py3k). - - On Python2, applications which need to ensure that ``bytes`` objects unpickle correctly under Py3k need must be changed to use a new type, ``zodbpickle,binary``. ``ZODB`` is configured with pickler / upickler from ``zodbpickle``, such that objects of this type will be pickled using the ``protocol 3`` opcodes for bytes (so that Py3k will unpickle them as ``bytes``). - - Existing data for the affected classes will need to be fixed up using a variation of convert_storages_. - - No changes to the ``ZODB`` packages on Py3k. Storages do not need to be configured with any custom pickle support. .. _convert_storages: Convert Database Storages + - - Need tool(s) to identify problematic data: - Classes which mix ``str`` and ``unicode`` values for the same attribute across records / instances. - - Utility which can apply per-class transforms to state pickles: - E.g., for instances of ``OFS.Image.Pdata``, convert the ``data`` attribute (which should be a Python2 ``str``) to ``zodbpickle.binary``. (Of course, these would probably be better off written out as blobs). - Or, for some application which mixes ``str`` and ``unicode`` under Python2 (either across instances or across transaction): upconvert any value of type ``str`` for the given attribute(s) to ``unicode``, using a configured encoding strategy (e.g, try ``utf8`` first, falling back to ``latin1``). - - One-time converter utility would use ``copyTransactionsFrom``-style pattern, opening the existing database readonly, getting pickles for each transaction, invoking the converter utility for each instance to fix up the pickle, then writing the converted pickles into the new database. .. _wrap_storages: Wrap Database Storages ++ - - A wrapper storage uses the converter utility (identified above) during the ``load`` operation, fixing up the object state it is handed to the instance's ``__setstate__``. - - During the ``save`` operation, the wrapper would fix up pickled instance state (after calling ``__getstate__``). - - Wrappers might be applied under Python2 (e.g., for apps where the databse is already converted to ``protocol 3``) as an alternative to replace_py2_cpickle_. - - Wrappers might be applied under Py3k (e.g., for apps where the databse is not already converted to ``protocol 3``) as an alternative to replace_py3k_pickle_.. Concrete Proposal - - I believe we will need to update ``zodbpickle`` and ``ZDOB`` to allow for any of the strategies to be applied. - - ``zodbpickle`` should provide the script which analyzes pickles in a database for inconsistent ``str`` / ``unicode`` usage. See: https://github.com/jimfulton/dbstringanalysis - - ``zodbpickle`` should provide the utility for registering per-class fixups. - - ``zodbpickle`` should provide the script which uses that utility do to one-time conversion of a storage (supporting convert_storages_). - - ``zodbpickle`` should provide a new ``binary`` type which Python2 applications can begin using to signal that attributes should be unpickled in Py3k as ``bytes``. See: https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes - - ``zodbpickle`` should provide a pickler/unpickler for use by Python2 clients who operate against converted storages (replace_py2_cpickle_). See: https://github.com/zopefoundation/zodbpickle/tree/py2_explicit_bytes - - ``zodbpickle`` should provide a pickler/unpickler for use by Py3k clients who operate against unconverted storages (replace_py3k_pickle_). See: https://github.com/zopefoundation/zodbpickle - - ``zodbpickle`` might need to provide a wrapper storage supporting straddle_no_convert_. Comments? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version
Re: [ZODB-Dev] Towards ZODB on Python 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/10/2013 09:19 AM, Jim Fulton wrote: Now, ZODB is built on top of pickles. And pickles in Python 2 know about two kinds of strings: str and unicode. But there are actually *three* kinds of strings in Python-land: * bytes * unicode * native strings (same as bytes in Python 2, same as unicode in Python 3) I hadn't encountered that term before. I see it informally used to refer to ``str``, which is bytes in Python 2 and Unicode in Python 3. This isn't a different kind of string. WSGI (PEP ) uses the term. In the case of pickles, we need to guarantee that the strings used for attribute names (including module attributes) are native strings. Oops, I see that you arrive there below: I should have read all the way to the bottom. Is this an issue for anything but names (object attributes and global names)? I don't think there's a native strings issue. There *does* seem to be an name issue. In Python 2 and Python 3, (non-buggy) unicode aware applications use bytes and unicode the same way, unicode for text, bytes for data. AFAICT, Python 3 has (admirably) changed the way names are implemented to use unicode, rather than ASCII. Am I missing something? This is a somewhat thorny, but still fairly restricted problem. I would hazard to guess that 99.923% of persistent classes pickle their state using their instance dictionaries. 99.9968% for regular Python classes. We know when we're pickling and unpickling instances and we can apply transformations necessary for the target platforms. I think the fix is pretty straightforward. In the default __setstate__ provided by Persistent, and when loading non-persistent instances: - On Python 2, ASCII encode unicode attribute names. - On Python 3, ASCII decode byte attribute names. The same transformation is necessary when looking up global names. Hmm, if zodbpickle has to handle the issue for non-persistent instances and global names, wouldn't it be simpler to make it handle persistent instances too? It can examine the stack inside 'load_dict' to figure out that the context is an instance, right? This will cover the vast majority of cases where the default __setstate__ is used. In rare cases where a custom setstate is used, or when Python 3 non-ASCII attribute names are used, then databases may not be sharable accross Python versions. Code with custom __setstate__ / __getstate__ where the difference matters is going to need porting anyway, so it might as well straddle. There is also likely to be breakage in dictionaries or BTrees where applications are sloppy about mixing Unicode and byte keys. I don't think we should try to compensate for this. Mixing None into a tree with bytes / text keys is likely a bigger problem than mixed bytes / text keys. These applications need to be fixed. One could write a database analysis script to detect this kind of breakage (looking for mixed string and unicide keys). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEUEARECAAYFAlE8pdoACgkQ+gerLs4ltQ7EswCeLZ0qq3INcYIxVNI55d6wurgy d5UAlA1jwoI1tWFwxfVp2ZhrO92Ctyk= =rPzb -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Towards ZODB on Python 3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 03/10/2013 11:55 AM, Jim Fulton wrote:
On Sun, Mar 10, 2013 at 11:25 AM, Tres Seaver tsea...@palladion.com
wrote:
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1
On 03/10/2013 09:19 AM, Jim Fulton wrote:
...
I think the fix is pretty straightforward.
In the default __setstate__ provided by Persistent, and when
loading non-persistent instances:
- On Python 2, ASCII encode unicode attribute names.
- On Python 3, ASCII decode byte attribute names.
The same transformation is necessary when looking up global
names.
Hmm, if zodbpickle has to handle the issue for non-persistent
instances and global names, wouldn't it be simpler to make it handle
persistent instances too?
No. It can't know when a key is going to be used for a persistent
attribute name.
It can examine the stack inside 'load_dict' to figure out that the
context is an instance, right?
Ugh. What stack?
The one where the unpickler keeps its work-in-progress?
static int
load_none(UnpicklerObject *self)
{
PDATA_APPEND(self-stack, Py_None, -1);
return 0;
}
static int
load_dict(UnpicklerObject *self)
{
PyObject *dict, *key, *value;
Py_ssize_t i, j, k;
if ((i = marker(self)) 0)
return -1;
j = Py_SIZE(self-stack);
if ((dict = PyDict_New()) == NULL)
return -1;
for (k = i + 1; k j; k += 2) {
key = self-stack-data[k - 1];
value = self-stack-data[k];
if (PyDict_SetItem(dict, key, value) 0) {
Py_DECREF(dict);
return -1;
}
}
Pdata_clear(self-stack, i);
PDATA_PUSH(self-stack, dict, -1);
return 0;
}
It would be much simpler to handle this in __setstate__ (or the
equivalent). This isn't exactly a lot of code.
Again, if we have to deal with non-persistent / global names inside the
pickler anyway, then why not do the same for persistent ones?
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlE8sSoACgkQ+gerLs4ltQ7veQCg1wIhHOo93Z3z3NTVG3tesJnS
GLAAoL5XNrzOR5oucmPcpDSNE3QJ7Kb9
=YUjX
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [zopefoundation/zodbpickle] 661e8b: Add minimal tests and Python ref. impl. for 'noloa...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 03/06/2013 06:29 AM, Marius Gedminas wrote: On Tue, Mar 05, 2013 at 05:05:36PM -0800, GitHub wrote: Branch: refs/heads/master Home: https://github.com/zopefoundation/zodbpickle Commit: 661e8b78f91a9e77da739c2f1d00780ad24d1efa https://github.com/zopefoundation/zodbpickle/commit/661e8b78f91a9e77da739c2f1d00780ad24d1efa Author: Tres Seaver tsea...@palladion.com Date: 2013-03-05 (Tue, 05 Mar 2013) Changed paths: M src/zodbpickle/pickle.py M src/zodbpickle/tests/pickletester.py Log Message: --- Add minimal tests and Python ref. impl. for 'noload'. Thank you, this made my day! BTW I've a plan for Python 3.2 compatibility in zodbpickle: move the new features (noload, bytes_as_strings) into _pickle_32.c, rename it to _pickle.c, then drop _pickle_33.c and the corresponding tests for 3.3-only features. We're not using any of the new pickle features introduced in Python 3.3, and I don't like to see an external package like zodbpickle using undocumented internal CPython APIs like _Py_IDENTIFIER. Sounds good to me. Note that the test coverage for the Python reference implementation is not great (and not just for 'noload*'). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlE3z4UACgkQ+gerLs4ltQ565wCgsk8K0qMqGA3FHUqXx3F5G09s 5D0AoJZep0kkI/S4mvxf8eBY9g62C5Tk =AIxH -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] ZODB Py3k porting status
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thanks to Marius and Stephan's efforts (with a little help from me), the 'py3' branch of ZODB now passes all unit tests, with most of the many-many ResourceWarnings now suppressed as well (we are down to the last two unclosed blob files). Today, I worked on making the branch work with Python 3.2. In order to support 3.2, I needed a 3.2-compatible verison of 'zodbpickle', as well as ZConfig. The ZConfig changes were trivial (avoiding u'' literals): I pushed out a 3.0.3 release with those changes. For 'zodbpickle', I created a second fork of '_pickle.c' (from the Python 3.2.3 release), and select between the two variants at build time based on the version of Python. I couldn't make a release there (Marius is the sole PyPI owner / maintainer), but was able to install it in my ZODB checkout's '.tox/py32' virtualenv for testing. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlEyp/kACgkQ+gerLs4ltQ7rowCeOsClMleFcDRFAYiNjaz6tyaJ wR4An2otbJ7c5KpRiV36jX24S6fJpADd =i/tT -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] API question
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 01/15/2013 06:11 AM, Jim Fulton wrote:
On Mon, Jan 14, 2013 at 1:32 PM, Tres Seaver tsea...@palladion.com
wrote:
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1
While working on preparation for a Py3k port, I've stumbled across
a fundamental issue with how ZODB structures its API. Do we intend
that client code do the following::
from ZDOB import DB, FileStorage db =
DB(FileStorage('/path/to/Data.fs'))
As Marius points out, this doesn't work.
or use the module as a facade ::
import ZODB db =
ZODB.DB(ZODB.FileStorage.FileStorage('/path/to/Data.fs'))
This doesn't work either. You haven't imported FileStorage.
WRT ZODB.DB, ZODB.DB is an age-old convenience. It's unfortunate that
ZODB.DB (the class) shadows the module, ZODB.DB, just like the class
ZODB.FileStorage.FileStorage shadows the modules
ZODB.FileStorage.FileStorage.FileStorage. (Of course, it's also
unfortunate that there's a ZODB.FileStorage.FileStorage.FileStorage
module. :)
If we had a do-over, we'd use ZODB.db.DB and
ZODB.filestorage.FileStorage, and ZODB.DB would be a convenience for
ZODB.db.DB.
I would actually prefer that clients explicitly import the
intermediate modules::
from ZDOB import DB, FileStorage db =
DB.DB(FileStorage.FileStorage('/path/to/Data.fs'))
So you don't mind shadowing FileStorage.FileStorage.FileStorage. ;)
or even better::
from ZDOB.DB import DB # This one can even be ambiguous now
FTR, I don't like this style. Somewhat a matter of taste.
Sure. I don't like using APIs via long, multi-dotted paths.
from ZODB.FileStorage import FileStorage db =
DB(FileStorage('/path/to/Data.fs'))
The driver for the question is getting the tests to pass under both
'nosetests' and 'setup.py test', where the order of module imports
etc. can make the ambiguous cases problematic. It would be a good
time to do whatever BBB stuff we need to (I would guess figuring out
how to emit deprecation warnings for whichever variants) before
releasing 4.0.0.
I'm pretty happy with the Zope test runner and I don't think using
nosetests is a good reason to cause backward-incompatibility. The
zope test runner works just fine with Python 3. Why do you feel
compelled to introduce nose?
I have a couple of goals:
- - I want 'python setup.py develop test' to work on a fresh checkout, mostly
because it makes doing multi-Python tests with tox simple.
- - Nose gives us easy access to Ned Batchelder's coverage package, which
allows me to get seriously good test coverage in place before trying to
port code to the straddle dialect.
- - I mean to keep the buildout + testrunner stuff working, too.
I'm sort of in favor of moving to nose to follow the crowd, although
otherwise, nose is far too implicit for my taste. It doesn't hande
doctest well at all.
Having said that, if I was going to do something like this, I'd rename
the modules, ZODB.DB and ZODB.FileStorage to ZODB.db and
ZODB.filestorage and add module aliases for backward compatibility. I
don't know if that would be enough to satisfy nose.
Likely so. I will give it a stab on the branch I'm working with.
I'm not up for doing any of this for 4.0. I'm not alergic to a 5.0
in the not too distant future. I'm guessing that a switch to nose
would also make you rewrite all of the doctests as unittests. As the
prrimary maintainer of ZODB, I'm -0.8 on that.
I'd like to see ZODB ported to Py3k ASAP, in a single codebase (as with
the ported 'zope.*' packages, 'persistent', 'transaction', and 'BTrees').
Trying to get there with doctests has seemed way too hard to me:
doctests are really fragile to cross-platform changes (especially various
'repr' changes).
In the case of the already-ported packages, I migrated most of their
doctests out of the software into testsed snippets inside Sphinx docs.
That move seems like a reasonable tradeoff: the examples still get
exercised as part of the docs, but they don't carry the weight of testing
the package.
Back to APIs, I think 90% of users don't import the APIs but set up
ZODB via ZConfig (or probably should, if they don't). For Python
use, I think the ZODB.DB class short-cut us useful. Over the last
few years, ZODB has grown some additional shortcuts that I think are
also useful. Among them:
ZODB.DB(filename) - DB with a file storage ZODB.DB(None)
- DB with a mapping storage ZODB.connection(filename) - connection
to DB with file storage ZODB.connection(None) - connection to DB
with mapping storage
More importantly:
ZEO.client us a shortcut for ZEO.ClientStorage.ClientStorage
ZEO.DB(addr or port) - DB with a ZEO client
ZEO.connection(addr or port) - connection to DB with a ZEO client
OK, let's keep DB-the-class at top-level scope, and rename the packages
(w/ BBB aliases).
Tres.
- --
===
Tres Seaver
[ZODB-Dev] API question
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
While working on preparation for a Py3k port, I've stumbled across a
fundamental issue with how ZODB structures its API. Do we intend that
client code do the following::
from ZDOB import DB, FileStorage
db = DB(FileStorage('/path/to/Data.fs'))
or use the module as a facade ::
import ZODB
db = ZODB.DB(ZODB.FileStorage.FileStorage('/path/to/Data.fs'))
I would actually prefer that clients explicitly import the intermediate
modules::
from ZDOB import DB, FileStorage
db = DB.DB(FileStorage.FileStorage('/path/to/Data.fs'))
or even better::
from ZDOB.DB import DB
# This one can even be ambiguous now
from ZODB.FileStorage import FileStorage
db = DB(FileStorage('/path/to/Data.fs'))
The driver for the question is getting the tests to pass under both
'nosetests' and 'setup.py test', where the order of module imports etc.
can make the ambiguous cases problematic. It would be a good time to do
whatever BBB stuff we need to (I would guess figuring out how to emit
deprecation warnings for whichever variants) before releasing 4.0.0.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlD0TycACgkQ+gerLs4ltQ4AgACg3MCYrEOga5KF8goWyu2OxjWe
H7QAoLEyHTShzBc9ZkMENWbG+hqzrpTg
=nBoy
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] API question
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/14/2013 03:33 PM, Marius Gedminas wrote: Can you demonstrate the ambiguity? As I've shown before, I was unable to find it, at least with Python 2.x. On the filesystem: $ find src/ZODB/FileStorage/ -name *.py src/ZODB/FileStorage/fspack.py src/ZODB/FileStorage/__init__.py src/ZODB/FileStorage/fsdump.py src/ZODB/FileStorage/fsoids.py src/ZODB/FileStorage/format.py src/ZODB/FileStorage/interfaces.py src/ZODB/FileStorage/tests.py src/ZODB/FileStorage/FileStorage.py So, 'ZODB.FileStorage.FilesStorage' could logically be either the 'Filestorage' module inside the 'ZODB.FileStorage' package, or else the same-named class. As with ZODB.DB, getting to the actual module is tricky, because both thses imports give you the class:: from ZODB.FileStorage import FileStorage and:: From ZODB.FileStorage.FileStorage import FileStorage I'm tempted to rename the 'DB.py' module 'db.py', and jam in a BBB entry in sys.modules for 'ZODB.DB'; likewise, I am tempted to rename the 'FileStorage.py' package 'filestorage', its same-named module '_filestorage.py', and jam in BBB entries for the old names. Those renames would make the preferred API: from ZODB import DB # convenience alias for the class from ZODB import db # the moodule from ZODB.db import DB # my preferred speling from ZDOB.filestorage imoprt FileStorage # conv. alias for class from ZODB import filestorage # the package from ZODB.filestorage import FileStorage # my preferred speling from ZODB.filestorage import _filestorage # if needed For extra bonus fun, we could rename 'ZODB' to 'zodb' :) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlD0oOgACgkQ+gerLs4ltQ4LOwCgu3VSRklLjFMdkuWLkUNV4h2S m/MAoKMI+ZrTqFUnXkgGNSw7Gq2yYN0V =67De -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Progress report: porting persistent, BTrees to Python3
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 12/13/2012 09:40 PM, Tres Seaver wrote:
I have gotten the trunk of persistent to build an test persistent's
extension modules under Python 3.2. There is a segfault under Python
3.3 which looks to be an actual Python bug:
http://bugs.python.org/issue16676
I will make a 4.0.5 release once I figure out what to do about that
test under 3.3 (maybe I'll just skip it).
I fixed the remainig issues in persistent and released 4.0.5 today: its
tests properly exercise the C extensions Under Python 3.2 / 3.3.
On the BTrees front: I have the extension modules compiling under
Python3 (the 'py3k' branch), but didn't read Lennart's excellent
guide closely enough to do the new module initialization dance
correctly. I plan to fix that tomoorow and hope to have the C
versions of the trees passing all the same tests as the pure-python
implementations. Once they pass, I will merge that branch to the
trunk and look to cut a new release.
Both packages are set up to test against all supported Python
versions using tox: at this point, that includes 2.6, 2.7, PyPy, 3.2,
and 3.3. The tox config also builds Sphinx docs and exercises the
doctest snippets, as well as running coverage (at this point only
under Python2).
I'm down to one test failure (out of 3600+) under Py3k: the test that
inserting an out-of-bounds long long key into an L*BTree. On Python 3.2,
the assertion fails::
==
FAIL: test64 (BTrees.tests.testBTrees.FamilyTest)
- --
Traceback (most recent call last):
File
/home/tseaver/projects/Zope/ZODB/BTrees-py3k/BTrees/tests/testBTrees.py, line
481, in test64
self.assertRaises(ValueError, s.insert, BTrees.family64.maxint + 1)
AssertionError: ValueError not raised by insert
On Python 3.3, the test fails with an OverflowError::
==
ERROR: test64 (BTrees.tests.testBTrees.FamilyTest)
- --
Traceback (most recent call last):
File
/home/tseaver/projects/Zope/ZODB/BTrees-py3k/BTrees/tests/testBTrees.py, line
481, in test64
self.assertRaises(ValueError, s.insert, BTrees.family64.maxint + 1)
File /opt/Python-3.3.0/lib/python3.3/unittest/case.py, line 571, in
assertRaises
return context.handle('assertRaises', callableObj, args, kwargs)
File /opt/Python-3.3.0/lib/python3.3/unittest/case.py, line 135, in
handle
callable_obj(*args, **kwargs)
File /opt/Python-3.3.0/lib/python3.3/unittest/case.py, line 153, in
__exit__
self.obj_name))
OverflowError: Python int too large to convert to C long
The latter one is easy, I think: we could just have the test assert
either ValueError *or* OverflowError. Figuring out why the insert
doesn't raise at all under 3.2 is too hard for my bleary eyes to tackle
tonight (I've already reverted one failed stab at it).
One good thing: the tests under Py3k now take about thes ame time as the
CPython 2.x ones (just over a minute on my box): that is down from over
five minutes before getting the C extensions working. The PyPy tests,
running without extensions, take about three minutes.
Onward-and-upward'ly,
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlDLybgACgkQ+gerLs4ltQ6VSwCgmwLLIEOArUzSsOLbMWnRhq2Q
a04AoIWVjuWqkYrSOo/3ukNL7rAiKskY
=yLOx
-END PGP SIGNATURE-
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Progress report: porting persistent, BTrees to Python3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have gotten the trunk of persistent to build an test persistent's extension modules under Python 3.2. There is a segfault under Python 3.3 which looks to be an actual Python bug: http://bugs.python.org/issue16676 I will make a 4.0.5 release once I figure out what to do about that test under 3.3 (maybe I'll just skip it). On the BTrees front: I have the extension modules compiling under Python3 (the 'py3k' branch), but didn't read Lennart's excellent guide closely enough to do the new module initialization dance correctly. I plan to fix that tomoorow and hope to have the C versions of the trees passing all the same tests as the pure-python implementations. Once they pass, I will merge that branch to the trunk and look to cut a new release. Both packages are set up to test against all supported Python versions using tox: at this point, that includes 2.6, 2.7, PyPy, 3.2, and 3.3. The tox config also builds Sphinx docs and exercises the doctest snippets, as well as running coverage (at this point only under Python2). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlDKkYwACgkQ+gerLs4ltQ5kcgCgqY8h7yLohuw4ukiYpiO0eeHZ HaQAnjVKDr4YcQtF+KKVD7tn/v+Lx+8f =T1HG -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 4.0.0a1 released
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/08/2012 05:24 PM, Alex Clark wrote: On 2012-11-08 14:16:48 +, Jim Fulton said: On Thu, Nov 8, 2012 at 4:21 AM, Marius Gedminas mar...@gedmin.as wrote: On Wed, Nov 07, 2012 at 04:45:59PM -0500, Jim Fulton wrote: Not to be confused with ZODB3! :) This is the first ZODB 4 release. Not to be confused with the old ZODB4 either, I suppose ;) http://wiki.zope.org/ZODB/ZODB4 How could anyone be confused by that? ;) Congrats on the release! FWIW, IIRC: pip install ZODB failed while easy_install succeeded. Not sure why. Yeah, just tried it again: - error: can't copy 'include/persistent/cPersistence.h': doesn't exist Interesting that easy_install doesn't barf. We don't any longer ship headers or C code in the ZODB, so that piece of the setup() should go. Patch attached. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCcXJkACgkQ+gerLs4ltQ4VIACfTR6Hey7c5kjJVA+MLEtU6Mzb 4WMAnAyn7dHA8EEtzYHQ6nlypaRETGrc =BHhe -END PGP SIGNATURE- Index: setup.py === --- setup.py (revision 128202) +++ setup.py (working copy) @@ -121,10 +121,6 @@ test_suite=__main__.alltests, # to support setup.py test tests_require = tests_require, extras_require = dict(test=tests_require), - # XXX: We don't really want to install these headers; we would - # prefer just including them so that folks can build from an sdist. - headers = ['include/persistent/cPersistence.h', - 'include/persistent/ring.h'], install_requires = [ 'transaction', 'persistent', ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/20/2012 01:47 PM, Jim Fulton wrote: I had the impression that Tres was proposing more. shrug I released BTrees 4.0.0, and created a ZODB branch for the (trivial) shift to depending on it: http://svn.zope.org/ZODB/branches/tseaver-btrees_as_egg/ That branch passes all tests, and should be ready for merging. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCC/WIACgkQ+gerLs4ltQ4c8QCeNSG1owrfJy1WRU3R9OEGTVBi Zx8An0PEnU0kpeYL1RkBlkQ5jfycw3vo =rDzU -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/14/2012 06:23 PM, Jim Fulton wrote: On Sun, Oct 14, 2012 at 6:07 PM, Laurence Rowe l...@lrowe.co.uk wrote: On 14 October 2012 22:49, Jim Fulton j...@zope.com wrote: On Sun, Oct 14, 2012 at 5:28 PM, Tres Seaver tsea...@palladion.com wrote: ... Well, I don't have time to chase BTrees. This could always be done in ZODB 5. :) I could help chop BTrees out, if that would be useful: most of the effort will be purely subtractive in the ZODB package (I don't think anything depends on BTrees). FileStorage uses BTrees for it's in-memory index. MappingStorage used BTrees. There are ZODB tests that use BTrees, but I suppose they could be fixed. I just don't think the win is that great in separating BTrees at this time. I don't think Hanno is suggesting removing BTrees as a dependency from ZODB but rather breaking out the BTrees package into a separate PyPI distribution to make it more visible to potential users outside of the ZODB community, e.g. http://www.reddit.com/r/Python/comments/exj74/btree_c_extension_module_for_python_alpha/ I had the impression that Tres was proposing more. shrug Nope. I just meant to break BTrees out and have ZODB depend on it. I thought is would be helpful to decouple the release process a bit, e.g. so that the pure-python BTrees stuff could move forward independently of the bigger ZODB release cycle. To do that, refactoring tests shouldn't be required. I guess it could be argued that the fsBTree should be part of the ZODB rather than BTrees distribution, but leaving it where it is would be much easier. Yup. If someone wants to do this in the next couple of days, it would be welcome. I really want to get these releases going. (And I have limited time.) I have split BTrees out to a separately-releasable project: http://svn.zope.org/BTrees/trunk/ If going forward with that is OK, I would be glad to do the work to: - Release BTrees - Update ZODB trunk to depend on it by the end of the weekend. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCBcqUACgkQ+gerLs4ltQ7peACcD/MaZw5p26Ac5S3mp0EO3gpd c1gAoMrfgkqxh+ouy0Y22ERHFa2ONDB3 =K6VQ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: ZODB 4.0 (without persistent)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/14/2012 05:20 PM, Jim Fulton wrote: On Sun, Oct 14, 2012 at 5:03 PM, Hanno Schlichting ha...@hannosch.eu wrote: On Sun, Oct 14, 2012 at 10:21 PM, Jim Fulton j...@zope.com wrote: Goal: Give persistent, ZODB and ZEO their own release cycles. In which distribution would BTrees end up in? Still ZODB. I think the pure-Python work for BTrees isn't yet finished, but I could be wrong. Not sure. But if we are extracting packages into separate distributions, we should move BTrees out as well. Well, I don't have time to chase BTrees. This could always be done in ZODB 5. :) I could help chop BTrees out, if that would be useful: most of the effort will be purely subtractive in the ZODB package (I don't think anything depends on BTrees). BTW, I expect to increase the tempo of ZODB and ZEO releases. Yay. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlB7LpAACgkQ+gerLs4ltQ6QbACgkLzap7zb/n0Bcc/b82AkYfG/ nPkAoIzF6jFO7lvr9lAN8dgLeIlN3ack =AZ3M -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/27/2012 09:55 AM, Hanno Schlichting wrote: On Mon, Aug 27, 2012 at 3:18 PM, Tres Seaver tsea...@palladion.com wrote: I think that was the attempt to convert a too-big number to a C 'long long'. I have adjusted the test to use 2**63 - 1 directly. Yep, tests pass here now. I will make a 4.0.2 release after to buildbots report success on Windows (assuming they do). The winbot only builds actual tagged releases. You can follow its progress at http://winbot.zope.org/builders/wineggbuilder - there's a run every 30 minutes. I'm confident enough in the fix, so I'd just cut a new release. OK, I'll make the release then. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA7h+kACgkQ+gerLs4ltQ7ALwCfY7nVKv1ZzXvYM3iduSwzCa37 iaQAn3xfaOnQfc+ZLkoT6lmQtPUPeMMM =eZGj -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/26/2012 07:05 AM, Hanno Schlichting wrote: On Sat, Aug 25, 2012 at 4:11 PM, Tres Seaver tsea...@palladion.com wrote: This failure, and the others like it, indicate that the buildout has failed to install persistent correctly (there is nothing Windows-specific about the failure):: I actually got the same error locally. The trouble is the timestamp.py module and the TimeStamp.c extension. During installation setuptools creates a wrapper for the extension file. So you end up with TimeStamp.so but also a timestamp.py wrapper. This wrapper overwrites the original timestamp.py module. You likely have a case-sensitive file system and therefor didn't get this problem. Whoever came up with the idea of a case-insensitive filesystem should be shot (or at least, whoever decided to keep doing that after moving off of DOS). I have worked around this problem (I think) by renaming the extension module to '_timestamp.so'. I released a 4.0.1 to PyPI with this fix, along with a fix for the int - long overflow on 32-bit systems. I think TimeStamp.c has been there before and is imported in other modules as from persistent.TimeStamp import TimeStamp. So we should keep that API and use a different name for the new timestamp.py module. I don't see the need for the BBB-preservation here: I documented the backward-incompatibility in CHANGES.txt instead. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA6c9IACgkQ+gerLs4ltQ6ocACeIqlGWazwos+yWGaMvgMCOkQD ZQoAn3oR2rY70a1oT6aTjLpofIVtVBNd =GzPs -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/25/2012 04:33 AM, Adam GROSZER wrote: Hello, I'm sorry to say, but the build (for r127209) before your merge was passing. See http://localhost:8009/builders/ZODB_dev%20py_265_win32/builds/750 This failure, and the others like it, indicate that the buildout has failed to install persistent correctly (there is nothing Windows-specific about the failure):: - - % --- Error in test test___delattr___p__names (persistent.tests.test_persistence.PyPersistentTests) Traceback (most recent call last): File c:\Python26_32\lib\unittest.py, line 279, in run testMethod() File c:\eggs\persistent-4.0.0-py2.6-win32.egg\persistent\tests\test_persistence.py, line 643, in test___delattr___p__names inst, jar, OID = self._makeOneWithJar() File c:\eggs\persistent-4.0.0-py2.6-win32.egg\persistent\tests\test_persistence.py, line 58, in _makeOneWithJar from persistent.timestamp import _makeOctets ImportError: cannot import name _makeOctets - - % --- The only solution I can think of is to delete those buildouts and start fresh. And there are more problems: http://winbot.zope.org/builders/persistent_py_265_32/builds/2/steps/test/logs/stdio all probably windows related. I don't have a clue how to diagnose this one. $ python 2.6 Python 2.6.8 (unknown, May 27 2012, 16:15:53) [GCC 4.6.3] on linux3 ... type(1073741697 * 1024) type 'int' What does that yield on the Windows box? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA43PUACgkQ+gerLs4ltQ4UrQCgzUc2GiwTd+FLu+UEJvCTeJSC mO4AoKy6rZvWHIVR5l9Cu6cpbLE5ajaP =Kgjg -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/ Note split of persistent.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 08/24/2012 10:05 AM, Tres Seaver wrote: On 08/24/2012 02:42 AM, Adam GROSZER wrote: Hello, I think this broke ZODB tests. http://winbot.zope.org/builders/ZODB_dev%20py_265_win64/builds/757/steps/test/logs/stdio On 08/21/2012 07:08 PM, Tres Seaver wrote: Log message for revision 127545: Note split of persistent. Changed: _U ZODB/trunk/ U ZODB/trunk/src/CHANGES.txt -=- Modified: ZODB/trunk/src/CHANGES.txt === --- ZODB/trunk/src/CHANGES.txt 2012-08-21 17:07:01 UTC (rev 127544) +++ ZODB/trunk/src/CHANGES.txt 2012-08-21 17:08:42 UTC (rev 127545) @@ -8,6 +8,9 @@ New Features +- The ``persistent`` package is now released as a separate distribution, + on which ZODB now depends. + - ZODB no longer depends on zope.event. It now uses ZODB.event, which uses zope.event if it is installed. You can override ZODB.event.notify to provide your own event handling, although Hmmm, those symbols are there: $ /opt/Python-2.6.8/bin/virtualenv /tmp/per4 ... $ /tmp/per4/bin/easy_install persistent ... $ /tmp/per4/bin/python -c from persistent.timestamp import _makeOctets I will try a fresh ZODB checkout and see if I can reproduce. I don't see any of those failures. I see only one failure in src/ZEO/tests/testZEO.py (ZEO.tests.testZEO.client_has_newer_data_than_server). It looks like a spurious doctest failure to me (the expected error is actually raised, but the rendered output doesn't match). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA3jsIACgkQ+gerLs4ltQ6vtACglapHfxTy0FV/V8fpTKkNcGpZ 0HYAnR9lKiNkc2yG1XjuJU9+DECvvtlz =V8FU -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Call for volunteers: help w finishing Python BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/04/2011 01:32 PM, Jim Fulton wrote: On Tue, Oct 4, 2011 at 11:36 AM, David Glick davidgl...@groundwire.org wrote: On 10/4/11 8:33 AM, Jim Fulton wrote: Someone recently told me I should be more agressive about asking for help. If someone is looking for an opportunity to help, finishing the Python version of BTrees would help a lot. I think I got this started pretty well, but ran out of time. This is needed for running ZODB on PyPy and jython, both of which I'd like to see. svn+ssh://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees Jim P.S. Much thanks to Tres for his work on the Python version of persistence. What tasks remain to be done? (I assume running the tests will give a starting point, but perhaps there are other todo items you know of?) Really, just getting the tests to pass. I think there are a lot of legacy, but still supporte features that need to be fixed. (This is a really old package.) In a fresh checkout of the branch, I see what looks like an infinite loop in the tests: I left it running for an hour just now, and it hung inside the '_set_operation' helper function inside the 'test_difference' testcase for 'PureOO' testcase. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlA0EKQACgkQ+gerLs4ltQ5XtwCfZ5LKg214I/I5/s3Pd+AtpgzI Ur0AnRIvmkBvuZKfZtlDg9mv1wObnUwX =Ocn+ -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: release persistent as a standalone package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/10/2012 08:52 AM, Tres Seaver wrote: On 06/30/2012 02:02 PM, Tres Seaver wrote: I would like to release a '4.0.0' version of the package, and switch the ZODB trunk to pull it in as a dependency (deleting the currently included (older) copy of persistent). One possible issue is that I have not (yet) made the C extensions work under Python 3.2: I don't know whether that should be a blocker for a release. I have a branch of ZODB which makes this go: http://svn.zope.org/ZODB/branches/tseaver-persistent_as_egg/ (note that the branch assumes 'persistent' can be installed from PyPI, or that you have separately installed it beforehand). In order to work around that fact that setuptools / distribute do not install headers when it installs eggs[1][2], I have tweaked the setup.py to pass a lazy string for the 'include_path' entry to allow BTrees to find the 'persisent/cPersistence.h' header. [1] http://bugs.python.org/setuptools/issue142 [2] https://bitbucket.org/tarek/distribute/issue/295 I released persistent 4.0.0 today to PyPI: http://pypi.python.org/pypi/persistent and updated my ZODB branch to use it http://svn.zope.org/ZODB/branches/tseaver-persistent_as_egg/?rev=127487view=rev At this point, I believe my branch is ready to merge to the trunk. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAmn70ACgkQ+gerLs4ltQ63qwCgqoG5ZE/X8HCgnlYuCU89OXEb XsgAn1PGyqY0or+6ifFvjvH40KMgp6g2 =lncl -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RFC: release persistent as a standalone package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have completed the work needed to make 'persistent' distributable as a standalone package. The effort (begun almost four years ago!) includes the following highlights: Platform Changes - - - Added explicit support for Python 3.2 and PyPy. - Note that the C implementations of Persistent, PickleCache, and Timestamp are not built (yet) on these platforms. - - Dropped explicit support for Python 2.6. - - Completed pure-Python reference implementations of 'Persistent', 'PickleCache', and 'TimeStamp'. Testing Changes - --- - - 100% unit test coverage. - - All covered platforms tested under 'tox'. - - Refactored some doctests as unittests. - - Removed all 'ZODB'-dependent tests: - Rewrote some to avoid the dependency - Cloned the remainder into new 'ZODB.tests' modules on the ZODB trunk. - - Added support for continuous integration using 'tox' and 'jenkins'. - - Added 'setup.py dev' alias (installs 'nose' and 'coverage'). - - Dropped dependency on 'zope.testing' / 'zope.testrunner': tests now run with 'setup.py test'. Documentation Changes - - - - Refactored many doctests as Sphinx documentation (snippets are exercised via 'tox'). - - Added 'setup.py docs' alias (installs 'Sphinx' and 'repoze.sphinx.autointerface'). I would like to release a '4.0.0' version of the package, and switch the ZODB trunk to pull it in as a dependency (deleting the currently included (older) copy of persistent). One possible issue is that I have not (yet) made the C extensions work under Python 3.2: I don't know whether that should be a blocker for a release. Comments? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/vPyMACgkQ+gerLs4ltQ4DsgCfRsKhBzvTWHehVI86I5sryhjF whQAoL3evyI6GKOfa7eDVjTUVh7B8Q+/ =82vd -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Call for volunteers: help w finishing Python BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/08/2011 06:31 PM, Pedro Ferreira wrote: Really, just getting the tests to pass. I think there are a lot of legacy, but still supporte features that need to be fixed. (This is a really old package.) If it's just a matter of making the tests pass, maybe this will help? Can you explain the rationale for dropping the use of '_marker' in favor of 'None'? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAk6gX10ACgkQ+gerLs4ltQ4XGwCfdm2XHVAyjuP1q3E3cfbwPisJ Cr0Al3yRi2sn3nNWLmf52Bv+cnppVng= =+ipL -END PGP SIGNATURE- ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] notes re trying to run ZODB on PyPy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/24/2011 08:02 PM, David Glick wrote: Alan Runyan asked me to post my notes from my attempt to get ZODB running on PyPy. It's very much an experimental work in progress (that I got distracted from), but hopefully this is at least useful to anyone else who wants to attempt the same thing. I first tried building ZODB trunk (with C extensions). Ran into the following issues: * PyPy is missing an implementation of the _Py_ForgetReference macro which is used by ZODB's persistent/cPickleCache.c * The check_argument_cmp check in BTrees/objectkeymacros.h fails for the root object of the database on PyPy * I got a PyPy exception re wrap_objobjargspec which appeared to be some issue with using the __setitem__ slot of an extension type At this point I gave up on trying to build the C extensions, and instead did the following: 1. Started with Jim's python-btrees branch (http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) 2. Replaced the 'persistent' package with a checkout of Tres' pure-Python persistent implementation (http://svn.zope.org/repos/main/persistent/trunk/persistent) 3. Set exts = [] in setup.py to disable all C extensions when installing. Next I worked around some issues in the pure-Python branches: * BTrees.___BTree.fsBucket class was missing toString and fromString methods, which I implemented. * persistent.picklecache.PickleCache class was missing update_object_size_estimation method (and indeed the whole limit-cache-size-by-bytes feature). I added it as a no-op stub. * persistent.picklecache.PickleCache.__setitem__ was raising a KeyError for a duplicate oid even when trying to set the same object already stored under that key * persistent.TimeStamp did not give the Python implementation, which I fixed by an import of persistent.timestamp + module alias if persistent.TimeStamp (the C module) is missing * persistent.pyPersistence.Persistent's __new__ needs to accept *args and **kw At this point I was able to start up the Pyramid ZODB scaffold and add an object to the DB root. Next I tried to get zodbshootout running as a benchmark. I didn't get this working, as I got stuck on an issue with the pure-Python PersistentMapping getting committed without its object state (appeared to be something to do with failing to get unghosted before it gets committed). Along the way, I noticed a couple incompatibilities with PyPy because PyPy implements the cPickle module as an alias to the pure-Python pickle module. This leads to some problems due to slight semantic differences between cPickle and pickle. In particular, cPickle.Pickler accepts a lone protocol argument; pickle.Pickler does not (leading to an error in ZEO.zrpc.marshal.encode). And pickle.Unpickler does not have a noload method, which ZODB.serialize.referencesf uses to obtain the referenced oids from a pickle without the overhead of unpickling. That's as far as I got. Hope this is helpful to the next guy. :) Cool, great work. Are the changes you needed to make available as patchers, or checked in? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5/RgAACgkQ+gerLs4ltQ7NmgCg2n4GT1wQWxPaRNWCguOtAb8q OQ4AoKcbDGep0ig5ZL/x+/m4hNeAVyuq =KtIZ -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] I would like append an indexer with uuid key at root of my ZODB, how can I append an function to commit event to update this uuid catalog ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/28/2011 05:21 AM, Stéphane Klein wrote: In repoze.catalog documentation, I read : Note that when you call index_doc, you pass in a docid as the first argument, and the object you want to index as the second argument. When we index the peach object above we index it with the docid 1. Each docid must be unique within a catalog; when you query a repoze.catalog catalog, you’ll get back a sequence of document ids that match the query you supplied, which you’ll presumably need to map back to the content object in order to make sense of the response; you’re responsible for keeping track of which objects map to which document id yourself. Url : http://docs.repoze.org/catalog/overview.html#indexing you’re responsible for keeping track of which objects map to which document id yourself. = it is this part I would like to implement. repoze.catalog ships with a 'DocumentMap' which can generate and track the docis for you. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAk4JypkACgkQ+gerLs4ltQ4i4QCXfmxdKLy2nJ3CgdZQeF+Li14F twCdGKTNvpv8A8/dn7ThUAlgjkURkNw= =50dg -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Large ZODB packing and progress
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/25/2011 07:17 AM, Shane Hathaway wrote: Whereas FileStorage packs by creating a new copy of the database, RelStorage deletes individual rows. Unfortunately, SQL delete statements tend to be surprisingly expensive. To get closer to what FileStorage does, RelStorage could theoretically copy rows into new tables instead of deleting rows. I've pondered doing that, but it would take some major work, I think. Maybe copying would be followed by truncating the original tables and re-populating them via INSERT FROM? Trse. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.co -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1nuhQACgkQ+gerLs4ltQ43PgCghmO5BM2qNznPdBUtuaoIcht8 v+QAn2usSMP84/K25Txl3sILWdtAV18y =UaLr -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate install problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/15/2011 11:37 AM, Chris Fanning wrote: On Tue, Feb 15, 2011 at 5:16 PM, Adam GROSZER agros...@gmail.com wrote: Hello, You got to use python 2.6, at least for zodbupdate. Thanks. I'd managed to get it going with 2.5 but I was getting this warning.. Warning: Missing factory for Persistence PersistentMapping Now, using python 2.6, I'm still getting the same warning. Looks like I need to get this missing factory because the resulting zodb.fs still gives the same error AttributeError: 'PersistentMapping' object has no attribute 'has_key' How do I install/configure this? You need to make the class importable at the old location in order to use instances of that class to convert to the new location. Something like:: import persistent.mapping import sys sys.modules['Persistence.PersistentMapping'] = \ sys.modules['persistent.mapping'] Then you should be able toload you old 'pages' object and create a new one (using the new class) and copy the contents into it from the old one. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1asnoACgkQ+gerLs4ltQ4zkwCgzodbX4J3d4Ig+bmk4Ld09k4j c40AnA9bOnrromUUb4ZI8tgicIUAHHVp =u1n2 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zodbupdate install problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/15/2011 12:06 PM, Tres Seaver wrote: On 02/15/2011 11:37 AM, Chris Fanning wrote: On Tue, Feb 15, 2011 at 5:16 PM, Adam GROSZER agros...@gmail.com wrote: Hello, You got to use python 2.6, at least for zodbupdate. Thanks. I'd managed to get it going with 2.5 but I was getting this warning.. Warning: Missing factory for Persistence PersistentMapping Now, using python 2.6, I'm still getting the same warning. Looks like I need to get this missing factory because the resulting zodb.fs still gives the same error AttributeError: 'PersistentMapping' object has no attribute 'has_key' How do I install/configure this? You need to make the class importable at the old location in order to use instances of that class to convert to the new location. Something like:: import persistent.mapping import sys sys.modules['Persistence.PersistentMapping'] = \ sys.modules['persistent.mapping'] Then you should be able toload you old 'pages' object and create a new one (using the new class) and copy the contents into it from the old one. Actually, you can install the 'Persistence' package directly (it is distributed separately from Zope): http://pypi.python.org/pypi/Persistence Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1as8MACgkQ+gerLs4ltQ7yzgCeKxXGqx/EIhxfR2BWDtZnOWpF qSoAn3T323/8RIS8wvYYRXmUX5n+lG2J =dRWP -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Request for merge: tseaver-squelch_2.6_warnings
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim, While doing work this morning on another branch, I came up with the following small changes which suppress unnecessary warning output when running the tests under Python 2.6: http://svn.zope.org/ZODB/branches/tseaver-squelch_2.6_warnings/ Please review and merge. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1S1IgACgkQ+gerLs4ltQ5rggCfeX5OqhdmvaFF9qkLMLEKm//v HtAAn2tgZ+AT92blJLYBNnKr1ST4zWHw =jTWs -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zeo server error message
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/31/2011 07:27 AM, Sylvain Viollon wrote: Hello, I have a customer who run a ZEO setup with ZODB 3.8.4. They report this error: 2011-01-31T12:02:24 Unexpected error Traceback (most recent call last): File /opt/zope/newbuildout/parts/zope2/lib/python/ZODB/ConflictResolution.py, line 207, in tryToResolveConflict resolved = resolve(old, committed, newstate) File /opt/zope/newbuildout/parts/zope2/lib/python/ZODB/ConflictResolution.py, line 141, in __cmp__ raise ValueError( ValueError: can't reliably compare against different PersistentReferences Does anyone know to what this error relate ? They have been running their ZEO setup for quite a while, and never experienced any issue. That doesn't seems to 'break' the database either. I am not really sure what to say about this error. I believe this is a real bug: the PersistentReference class is raising a ValueError becuase the object being compared is not itself, which implies an irresolvable conflict: it should be raising ConflictError, or else a custom error which would be caught by 'tryToResolveConflict' and converted into a ConflictError. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1G4aYACgkQ+gerLs4ltQ7LMQCdG1u1RF2kWZbhkgwkBbfKTkNy U9wAoMYQ0Tv/SgiXNlCxuUDZaJWjWAnU =PH8G -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Bug (?) in zope/publisher/publish.py:unwrapMethod
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/25/2011 12:19 PM, Vincent Pelletier wrote: Hi. In publish.py[1], unwrapMethod tried to find what can be used to publish an object. In a site, I had someone create a very-badly-named func_code external method in a place accessible by acquisition from every page on the site (this bad by itself, and I corrected it already). This caused unwrapMethod to think it can use any object directly for publishing, because of: elif getattr(unwrapped, 'func_code', None) is not None: break and unwrapped is still in an acquisition context. Shouldn't the checks be done on unwrapped (from acquisition context) objects instead, to prevent such stupid mistake to have such a wide impact. I have the intuition that this could even be a security problem, allowing an unexpected object to be called instead of another, but I cannot come out with an example. Do you think there is anything to fix in zope.publisher ? If so, I'll open a bug. [1] http://svn.zope.org/zope.publisher/trunk/src/zope/publisher/publish.py?view=markup The 'zope.publisher' package doesn't know anything about acquisition-wrapped objects: fixing this issue there would require adding either a dependency on 'Acquisition' (a bad idea, I think) or else adding logic like the 'im_func' bit which peeled away 'aq_self' until all wrappers were gone (using knowleged of Acquisition's API / semantics without importing it). The Zope2-specific version of 'mapply()' (in 'ZPublisher.mapply') is the right place to fix this issue, if it is to be fixed: it could ensure that the acquisition wrapper was stripped away before calling 'unwrapMethod()'. P.S. This issue is off-topic for the ZODB list: I have cross-posted to 'zope-dev': please follow up there. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0/H5sACgkQ+gerLs4ltQ4jdQCgzI+yNAuaX9cqPg77FLHxbzUG abwAn0BJoQ5ImlTrocRMY3KT3M2royOz =WSyR -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Lazy Instantiation
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/22/2011 08:54 AM, frits.swinkels wrote: I am new to ZODB and to this list: greetings. I need to get some points clear about precisely when objects are fully instantiated in memory. It may be a simple matter of pointing me at some documentation; I read pretty much anything I could find but there may a problem of your jargon being different from mine:) The main data structure of my application is a directed graph of nodes; each node has at least three slots which contain other nodes (e.g. parent, right child, left child; the parent always exists, the child slots may be None). I only want to instantiate that part of the graph that I am working on. Q1. I assume that when an object is instantiated, its sub objects will be in the form of some kind of proxy (perhaps that ghost I met in the ZODB How document?). Is this assumption correct? Yes. The ghost has not had its instance dictionary populated yet. Attempting to access any instance attribute will unghostify the object. Q2. Given an object, designated by the variable a, in which of the following statements does the child become instantiated (activated?) ? child = a.leftchild name = child.name The second (assuming that the nodes in your data structure derive from a persistent base class, such as Persistent, or one of the BTrees), Q3. Actually, before the child is instantiated, I somehow would like to query the object a, whether there is anything in the slot, i.e. whether the content of the slot is not None. Of course, this query should not trigger the instantiation. That should already be working, again assuming your objects derive from Persistent. I had the application running in another OO language. An object would first carry a proxy with the OID in a slot; it would get instantiated on access. The ad-hoc database made use of the Reiser file system, which allowed indexing serialized objects on their OID in a B tree. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk065S4ACgkQ+gerLs4ltQ510wCfSujRaHMG7utqgsgRbEEHwmiI iFsAoKEv4K5tIPYR2su/o3aEUDyVfTta =2Jam -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB packing and possible side effects?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/09/2010 05:32 AM, Mark Van den Borre wrote: I'm trying to read up on ZODB packing and possible side effects. Does anyone know about comprehensive docs on this? Think interaction with repozo backup processes, a pack removing weakly referenced objects, impact on the Data.fs index, ... - - Repozo detects packs, and forces a full backup after one, even if you request an incremental. - - You need to explain what you mean by weakly referenced. - - Pack causes the index to be written. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0A90oACgkQ+gerLs4ltQ6tKwCfdd/XWm5zsYhuEUid7NxCmS1W sLEAoNI6YRioiukHoV7kEtDSGM+ifxsS =u09G -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storing ordered elements
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/02/2010 01:14 PM, Pedro Ferreira wrote: OOSets are ordered. Yes, I know. My problem was that I wanted to order the elements by timestamp, and I didn't want to wrap anything around them just to have them ordered. But I think that in the end that's what I will do... or just make them comparable by timestamp. One alternative would be to store them as a tuple, (timestamp, object), which should be super-cheap, and still allow for normal comparisons of your domain objects. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzQTIoACgkQ+gerLs4ltQ40ywCeKZ34CiZ5889xZ4QeM0nKGNYT nKgAoM8V+9nXjY6aG+3wQrq6GhbXW4bg =UHQx -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fstest missing as script
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Mon, Aug 30, 2010 at 9:45 AM, Adam GROSZER agros...@gmail.com wrote: Hello, Is it intetional that fstest.py is missing from [console_scripts]? Anyone against adding it? Many of the scripts in ZODB and ZEO were one-offs that should never have been checked in. Most are not tested and many are not well thought out. Eventually, I'll find time to go through them all and remove ones that should be removed and whip others into shape. For this reason, I object to adding fstest as a console script. That objection seems a little odd, given the following checkin message for the file: $ svn log -r 71254 $ZSVN/ZODB/trunk/src/ZODB/scripts/fstest.py r71254 | jim | 2006-11-21 17:01:52 -0500 (Tue, 21 Nov 2006) | 3 lines Moved the scripts down into subpackages to they can be installed via entry points. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkx74+AACgkQ+gerLs4ltQ7x8QCffSVpbFxTJ5cyDRjRO7mydpar NnwAoJl6OCE3IBKtgEkibAIn4TKVgYiE =2u8C -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: On Mon, Aug 16, 2010 at 12:14 PM, Pedro Ferreira jose.pedro.ferre...@cern.ch wrote: Could this be some problem with using persistent objects as keys in a BTree? Some comparison problem? I'm not entirely sure about this, but I think using persistent objects as keys isn't supported. Looking at the code, I doubt using anything expect simple types like unicode strings or tuples of simple types will work without further work. From what I can see in the code, BTree's use functions like PyObject_Compare to compare different keys. Persistent doesn't implement any special compare function and falls back to the standard hash algorithm for an object. This happens to be its memory address. The memory address obviously changes over time and the same address gets reused for different objects. I think implementing a stable hash function for your type could make this work though. The ZODB gods correct me please :) Btrees require comparability, rather than hashability: your persistent type needs to define a total ordering[1], which typically means defining '__cmp__' for your class. You could also define just '__eq__' and '__lt__', but '__cmp__' is slightly more efficient. [1]http://www.zodb.org/documentation/guide/modules.html#total-ordering-and-persistence Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkxpK18ACgkQ+gerLs4ltQ5gIwCfQDXxLwxpWrghFtRF76MBxesS RtcAnA3+yv+6+XfDRkPBBZcZpoqZ6mO0 =ygVN -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Merge request: ZODB/branches/tseaver-better_repozo_tests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tres Seaver wrote: Jim Fulton wrote: On Fri, May 14, 2010 at 1:57 PM, Tres Seaver tsea...@palladion.com wrote: ... Jim, would you please review and merge the 'tseaver-better_repozo_tests' branch to the ZODB trunk? The only non-test change is the one I'm replying to, which is intended purely to make stuff more unit testable. Done Once this merges, I can do the index saving work on top of the new tests. Cool. Thanks. I have landed that feature on a new branch: http://svn.zope.org/ZODB/branches/tseaver-repozo_index/ Please review and merge, if ready. Note that the index restoration feature cannot work in the default case (no output file specified, ergo data written to stdout). Merged. Thanks! Note that the index restoration feature cannot work in the default case (no output file specified, ergo data written to stdout). I wonder if anyone uses the write-to-standard-output mode. It seems like an attractive nuisance. It enables some kinds of sysadmin-y solutions, like doing the backup over SSH when you don't have disk space on the host machine. We might make it not be the default, and require that you spell it '-o -' or something. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvxlEsACgkQ+gerLs4ltQ6ffQCeNfst5goiks18P7bGHtPgqJTm FwEAmgOdQnMIZHcPXOZzCnIx3X8cTGV1 =99fp -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Merge request: ZODB/branches/tseaver-better_repozo_tests
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Fri, May 14, 2010 at 1:57 PM, Tres Seaver tsea...@palladion.com wrote: ... Jim, would you please review and merge the 'tseaver-better_repozo_tests' branch to the ZODB trunk? The only non-test change is the one I'm replying to, which is intended purely to make stuff more unit testable. Done Once this merges, I can do the index saving work on top of the new tests. Cool. Thanks. I have landed that feature on a new branch: http://svn.zope.org/ZODB/branches/tseaver-repozo_index/ Please review and merge, if ready. Note that the index restoration feature cannot work in the default case (no output file specified, ergo data written to stdout). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvu2fwACgkQ+gerLs4ltQ45/ACeNQm49PSe862DLfECnGgsTP3w b/0An3Gl65Pt7YKE3lJQh9xkM1gFytFq =bdIy -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo full backup and index files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: On Fri, May 14, 2010 at 10:24 AM, Christian Theune c...@gocept.com wrote: Hmm. If the full backup is just a regular FS file then you could start with the naive approach and just open/close it once after performing a backup as that would create the index file. Sure. That would be an easy but also rather inefficient way. In my case it takes only about 15 minutes of rather heavy I/O to create the file, but for larger files this time usually goes up. From what I understand repozo does the following: 1. it opens the real file storage file as a read-only FileStorage 2. calculates the byte position of end of the last complete transaction 3. closes the file storage 4. opens the fs file as a normal binary file 5. copies over all bytes up to the calculated position into a temp file 6. closes the fs file and temp file and renames the temp file according to some timestamp What I'm wondering is, if you could copy or otherwise create the index as part of the second step. In step 5 we only deal with bytes and cannot parse those, but in step 2 we have a proper ZODB FileStorage object and can to some degree decide which transaction we want. This strategy should be workable for both full and incremental backups: the storage opened in step 1 has an attribute, '_index', which is an fsIndex instance. We should be able to save the index into the repository direcotry with a timestamped filename, something like:: index_file = os.path.splitext(dest)[0] + '.index' fs._index.save(fs._pas, index_file) At restore time, we would look for the index corresponding to the full or incremental being restored, and copy it into place followi[ng the successful copy of the data. As usual, the tedious part about implementing this feature will be adding / updating tests. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvtV0EACgkQ+gerLs4ltQ4JFgCfYZ3PV4pZCP7NInG1lW36F0ke sRMAoL24vUmOIpb2W2XOps7I9TuxmGRk =aMi4 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] repozo full backup and index files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Tres Seaver wrote: Hanno Schlichting wrote: On Fri, May 14, 2010 at 10:24 AM, Christian Theune c...@gocept.com wrote: Hmm. If the full backup is just a regular FS file then you could start with the naive approach and just open/close it once after performing a backup as that would create the index file. Sure. That would be an easy but also rather inefficient way. In my case it takes only about 15 minutes of rather heavy I/O to create the file, but for larger files this time usually goes up. From what I understand repozo does the following: 1. it opens the real file storage file as a read-only FileStorage 2. calculates the byte position of end of the last complete transaction 3. closes the file storage 4. opens the fs file as a normal binary file 5. copies over all bytes up to the calculated position into a temp file 6. closes the fs file and temp file and renames the temp file according to some timestamp What I'm wondering is, if you could copy or otherwise create the index as part of the second step. In step 5 we only deal with bytes and cannot parse those, but in step 2 we have a proper ZODB FileStorage object and can to some degree decide which transaction we want. This strategy should be workable for both full and incremental backups: the storage opened in step 1 has an attribute, '_index', which is an fsIndex instance. We should be able to save the index into the repository direcotry with a timestamped filename, something like:: index_file = os.path.splitext(dest)[0] + '.index' fs._index.save(fs._pas, index_file) At restore time, we would look for the index corresponding to the full or incremental being restored, and copy it into place followi[ng the successful copy of the data. As usual, the tedious part about implementing this feature will be adding / updating tests. I would be willing to make a stab at this, if we can hold off on a 3.10.0 beta until I've had a chance to try it. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvtX0gACgkQ+gerLs4ltQ4dWQCgs3Icg+omT7O3xfxCPoIiIyCm zuEAoKprV5ggwEx9xZHC7bWVX5Zfuogc =AKk0 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction 1.1.0 released
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: I've made a new transaction release with the changes we've discussed recently, as well a threading-related bug fix. The retry iterator goodness won't work with ZODB until the next ZODB release (after 3.10.0a2). Great! BTW, I had asked the owner of the 'transaction' project on Launchpad to transfer ownership to the 'zope-admin' group. He did so, and I have configured the project to track its own bugs, as well as updating the metadata, etc. I made the 'zodb-developers' group the driver for the project. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvsHjQACgkQ+gerLs4ltQ68FwCfZtesBqPZUZDpZwSiSxM1don5 FfgAoIuy3mTM7P2tfVrwMIXJ/H2bnmbB =gbdp -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Plone-Users] question about using repozo and blob storage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ralf Hemmecke wrote: Subject: Re: [Plone-Users] question about using repozo and blob storage Date: Fri, 7 May 2010 10:05:59 -0700 (PDT) From: ajung li...@zopyx.com To: plone-us...@lists.sourceforge.net This list is the wrong place for such deep ZODB related discussions. Suggest this on the zodb-dev list. -aj Ralf Hemmecke-2 wrote: On 05/07/2010 06:40 PM, ajung wrote: Blobs are - as David explained - a _core_ component of the ZODB. Content stored through the file storage or blob storage are handled and treated in the same way to the application. If you want to write a SCM-based backend for ZODBI am not sure how far you will get and if it would be reasonably fast and without major architectural issues. I understand that meta data lives still in ZODB. And I wouldn't want to suggest that people deal with the blobs in any other way than through plone. One should only be able to have read access to the blobstorage (if it's not through plone, otherwise disaster may happen). I haven't actually looked into how blobs are actually stored on the file system, i.e., in particular what name they will have and whether the file system structure is the same as the folder hierarchy inside plone. But that doesn't actually matter. I assume that they are stored in some directory in a directory for blobs, let's call it blobdir. I speak now for git, but other SCMs are quite similar (maybe not svn). This blobdir is turned into a repository by simply saying git init. Then when plone stores a file, it will write it into (a subdir of) blobdir. With GIT behind that should read: (1) write it into blobdir and then (2) git add FILENAME (3) git commit -m some message That's it. If the user uploads a new file to plone, then plone doesn't have to care about renaming the old file and storing the new, it simply has to do THE SAME steps (1)..(3) as above. I call that easier. Maybe packing is a bit more difficult and depends on the SCM in question. For git it would be a simple git filter-branch (of course with appropriate parameters) followed by git gc. Just my two cents. I would find such a GIT backend wonderful. But I've no experience with how that blob stuff is actually done through plone/zope. So sorry if I cannot provide any code. Sharing the blobs using a DVCS pull sounds like an interesting approach. One flaw with your outlined implementation is that ZODB blob files are *never* updated: rather, if a ZODB object updates the blob object, it creates a new blob file and stores its ID. Old blob files are cleaned up during a pack. If you want to experiment with the idea anyway, then writing a wrapper storage is probably the way to go: DemoStorage is one such beast, as is the BlobStorage itself. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvoTrgACgkQ+gerLs4ltQ5HBgCgwXNqzE4KYWAQZYEJMPSoi18t 09EAniD4ucOThPmJRJ8A9TjmVuQv3gQV =QUON -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Plone-Users] question about using repozo and blob storage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Mon, May 10, 2010 at 2:21 PM, Tres Seaver tsea...@palladion.com wrote: ... If you want to experiment with the idea anyway, then writing a wrapper storage is probably the way to go: DemoStorage is one such beast, as is the BlobStorage itself. I don't consider the wrapper approach to have been very effective for BlobStorage, although it was a reasonable thing to try at the time. BlobStorage is likely to go way. (This comment should not be construed as supporting the idea of implementing some sort of blob-storage back end using DVCS.) I'm having troube with the last sentence: do you really mean you don't like the idea of blobs-in-DVCS? Or did you drop the second part of a double negative inadvertently? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEUEARECAAYFAkvodNQACgkQ+gerLs4ltQ6qYQCfSUDolO7AgmwnpsE75oYSAYVT 4mAAmMGLaO/HvEEwsGLtxYV4U8vevBk= =2VUg -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Plone-Users] question about using repozo and blob storage
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ralf Hemmecke wrote: Tres Seaver tseaver at palladion.com writes: Sharing the blobs using a DVCS pull sounds like an interesting approach. One flaw with your outlined implementation is that ZODB blob files are *never* updated: rather, if a ZODB object updates the blob object, it creates a new blob file and stores its ID. Old blob files are cleaned up during a pack. Interesting. You know what git does? If one adds a file X (actually git deals with contents not files) to the repository, git computes a hash value (sha1) and uses that value as the name for the file. It's stored under the .git directory. Some update of bookkeeping data and that's it. If the contents of the file X changes, and it is committed to the git repository, git computes a sha1 and stores the new content under that new sha1 name + updating of the bookkeeping. The old file is not deleted. Git does not even bother to compress anything. That would be done by an additional git gc. In contrast to most other SCMs git can be considered as a database. That git is mostly recognized as a version control system, is only half of the story. The underlying storage system is what make git so interesting. I have no good knowledge of zodb and you are certainly doing great with it, but my concern is with the blobs (which as I hopefully understand correctly are to be stored outside of Plone's Data.fs). Storing versions of files into a DVCS is like freeing the zodb (for blobs) from doing its own versioning. Actually, I think conceptually git and the zodb are quite similar. One transaction is nothing else than a commit in the SCM sense. If you want to experiment with the idea anyway, then writing a wrapper storage is probably the way to go: DemoStorage is one such beast, as is the BlobStorage itself. I'm sorry, I cannot do this. I've simply no time for opening up another area of development for me. I understand that completely: I'm there myself. ;) Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvodS4ACgkQ+gerLs4ltQ68JACgnReABZhAcIOcpxADMftidN6D WK4AniKQUv7LU1X1fSqFY8gFZ3xuhllA =J5Q0 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Ever-Increasing Memory Usage (even with cache-size-bytes)
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jim Fulton wrote:
On Mon, May 10, 2010 at 3:27 PM, Ryan Noon rmn...@gmail.com wrote:
snip
Here's my code:
self.storage = FileStorage(self.dbfile, pack_keep_old=False)
cache_size = 512 * 1024 * 1024
self.db = DB(self.storage, pool_size=1, cache_size_bytes=cache_size,
historical_cache_size_bytes=cache_size, database_name=self.name)
self.connection = self.db.open()
self.root = self.connection.root()
and the actual insertions...
set_default = wordid_to_docset.root.setdefault #i can be kinda
pathological with loop operations
array_append = array.append
for docid, wordset in docid_to_wordset.iteritems(): #one of my
older sqlite oodb's, not maintaining a cache...just iterating (small
constant mem usage)
for wordid in wordset:
docset = set_default(wordid, array('L'))
Note that you are creating the array willy-nilly in the inner loop here.
I would nearly always write that as::
docset = wordid_to_docset.root.get(wordid)
if docset is None:
docset = array('L')
wordid_to_docset.root[worid] = docet
array_append(docset, docid)
Why are you using an unbound method here? The following would be
clearer, and almost certainly not noticeably slower:
docset.append(docid)
n_docs_traversed += 1
if n_docs_traversed % 1000 == 1:
status_tick()
if n_docs_traversed % 25000 == 1:
self.do_commit() #just commits the oodb by calling
transaction.commit()
Don't forget the final commit. ;) Also, I don't know what the 'array'
type is here, but if it doesn't manage its own persistence, then you
have a bug here: mutating a non-persistent sub-object doesn't
automatically cause the persistent container to register as dirty with
the transaction, which means you may lose changes after the object is
evicted from the RAM cache, or at shutdown.
snip
Also note that memory allocated by Python is generally not returned to
the OS when freed.
Python's own internal heap management has gotten noticeably better about
returning reclaimed chunks to the OS in 2.6.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkvoeIoACgkQ+gerLs4ltQ6WjACgsvDmG96nD2iPl/noiHS5ThdL
SdIAn1Ei+yfzRyJ4W1lwvuThBj9BxzGt
=nrBB
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Using zodb and blobs
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nitro wrote: After lots of googling and browsing the source I can answer some of my questions: - What's the difference between storing bigger objects as blobs and as plain large strings? Plain large strings cannot be streamed for instance. Products like Zope chop up their file uploads into 64kb chunks which are then stored as individual objects in the zodb. That was the strategy before blobs. ZODB versions since 3.8 support storage of BLOb data as files on the filesystem. - Can I stream in parts of a blob/large string without having to read all of it? I can get a file handle to a blob. Strings are always read as a whole. - Where can I find example code on zodb blobs? E.g. how do I save a blob, how do I read it back in? The ZODB/tests directory features a few blob doctests which provide all the necessary code to get started. Having this on zodb.org would be nice (especially since the doctests are already ReST-formatted). Additionally I made some quick performance tests. I committed 1kb sized objects and I can do about 40 transaction/s if one object is changed per transaction. For 100kb objects it's also around 40 transactions/s. Only for object sizes bigger than that the raw I/O throughput seems to start to matter. 40 tps sounds low: are you pushing blob content over the wire somehow? Still don't know the answers to these: - Does it make sense to use ZODB in this scenario? My data is not suited well for an RDBMS. YMMV. I still default to using ZODB for anything at all, unless the problem smells very strongly relational. - Are there more complications to blobs other than a slightly different backup procedure? You need to think about how the blob data is shared between ZEO clients (your appserver) and the ZEO storage server: opinions vary here, but I would prefer to have the blobs living in a writable shared filesystem, in order to avoid the necessity of fetching their data over ZEO on the individual clients which were not the one pushing the blob into the database. - Is it ok to use cross-database references? Or is this better avoided at all cost? I would normally avoid them out of habit. They seem to work, though. And new questions: - Does the _p_invalidate hooking as outlined at http://www.mail-archive.com/zodb-dev@zope.org/msg00637.html work reliably? Never tried it, nor felt the need. - Are there any performance penalties by using very large invalidation queues (i.e. 300,000 objects) to reduce client cache verification time? At a minimum, RAM occupied by that queue might be better used elsewhere. I just don't use persistent caches, and tend to reboot appservers in rotation after the ZEO storage has been down for any significant period (almost never happens). From what I've read it only seems to consume memory. Note that the ZEO storage server makes copies of that queue to avoid race conditions. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvDd5sACgkQ+gerLs4ltQ5h4wCghqTQsNO/5XrqHMZyhi8Hz17d oRcAn1el86604KoTTWB8Bx5R13ZlvQB/ =momg -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Testing gocept.zeoraid
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Sylvain Viollon wrote: I am currently testing gocept.zeoraid, and this seems to work fine. However, I have few questions: - On the pypi the last released version is 1.0b7. In the Zope SVN it seems to be only 1.0b1. Hmm, I see tags for 1.0a1, b1-b7:: $ svn ls $ZSVN/gocept.zeoraid/tags 1.0a1/ 1.0b1/ 1.0b2/ 1.0b3/ 1.0b4/ 1.0b5/ 1.0b6/ 1.0b7/ and the trunk changelog shows 1.0b8 as unreleased:: $ svn cat $ZSVN/gocept.zeoraid/trunk/CHANGES.txt | head Change History == 1.0b8 (unreleased) -- - #484437: Recipe handles address in zeo.conf incorrectly. When giving\ the ZEO address on the server only as a port that port would be ignored\ resulting in 127.0.0.1:8100 to be assumed for the management scripts. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkungIwACgkQ+gerLs4ltQ4bSwCfXumBlBylXUIpt1kJC7s85StG o6wAn1IyqebgEroNngKJ5jg06A7qx0XS =CwE5 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Merge request: repozo cleaning up old files
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim, I would like to see this branch merged into a release of ZODB: $ZSVN/ZODB/branches/tseaver-lp143158-feature/ The branch fixes the tests for repozo to make them true unit tests, as well as adding a long-missing feature (optionally cleaning out old backup sets when making a new full backup). Ideally, I would prefer that it go into a new 3.9.x release, since 3.9.x is the version most people are still using in production. If the risk seems too high (I think any risk here is very negligible), then I would at least like to see it in a 3.10.x release. Thanks! Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkuZYOoACgkQ+gerLs4ltQ7qoQCgl69xbHzZkvzs0mGcGF3ilAGo rqcAnArLyAevGzu7zNtYCQEABjpa76/w =JL48 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Location of transient caches
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andy Theyers wrote: Hi all We have a Debian server running 3 Zope instances, each mounting its database from a single ZEO instance on the same server. Until very recently we had been running each Zope with a persistent cache (using the 'client' directive in the zeo_client stanza of zope.conf). Each cache was 300MB, as recommended by various buildout scripts and other how to documents I had read. While under this configuration the site failed due to cache corruption. This caused us to remove the persistent cache option, again as recommended in various mailing list posts. However, as we are using the default Debian disk partitioning (whereby / is only 1GB) we ran out of disk space, as the transient caches appear to be being created in /tmp Is there any way I can change the location of the transient cache files, putting them on a partition with more space? I've been scouring the source but am yet to come up with the bit that creates the temporary cache files. It's not an area of Zope that I've got any experience of. Any help would be greatly appreciated. The cache uses the 'tmepfile' module, which uses a number of environment variables to select the location: http://docs.python.org/library/tempfile.html Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt+tUEACgkQ+gerLs4ltQ4uHgCeOIV05O6lwvsUdeiOnUpgekoc Zi0AoJvZ1zCbAO0ehx25uQX/krgvJ7X0 =IyXE -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Multi-Threaded Access
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jared Smith wrote: I'm looking at using ZODB and have two different databases stored in different files where different threads will be periodically insert/update/delete various records in both databases. Couple of questions to make sure this is feasible: When one of the threads wants to do an insert of a set of records I need to ensure that those changes are submitted in a transaction that does not overlap with the records being inserted in another thread. Looks like the default API assumes there is a single global transaction (transaction.commit()) however I saw mention of creating your own transaction.manager that you can bind at DB.open() time. Does this mean to support this model I have to re-open the database and do the binding of a transaction manager in every method where I want to do this transactional action? For the moment I was assuming a DB access object that is shared (singleton access class). Is the better model to have something like a pool of DB access classes where each one maintains a DB handle and a transaction and one of the access classes would be taken out of the pool and used every time a transaction needs to be run? Worried a bit about the overhead of re-opening the DB is this not something to worry about? This second pattern is the one commonly used in Zope and other ZODB-based web apps: each request gets a connetion object from a pool maintained by the database, and uses the objects in its cache to satisfy the request. If the request completes successfully, the application then commits the transaction, other wise it aborts. In either case it returns the connection to the pool at the end of the request. The current transaction for a request is stored in a thread local. Committing a transaction can fail (raise a ConflictError) if another, overlapping transaction has modified some of the same objects: in this case, the application thread has the option of synchronizing its connection's cache and retrying the request (Zope tries up to three times). Hope that helps, Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktiemEACgkQ+gerLs4ltQ4jVgCeLUC3nyLPToguuFujL8fe06Jg 8T0AoIR8ShyGrHDiq41DsmGBKgLs+KxV =Rm/D -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persistent object has empty __dict__ for a little while
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Martin Aspeli wrote:
Hi,
This one is pretty high no the list of weirdest things to have happened
to me in a while. Basically, I have a persistent object that has an
empty __dict__() on the first request, until it suddenly decides to have
data again.
I'm on ZODB 3.9.3, using Zope 2.12 and Plone 4. I have a persistent
object, /plone/portal_registry:
class Registry(registry.Registry, SimpleItem):
pass
The base class is:
class Registry(Persistent):
def __init__(self):
self._records = Records(self)
Records is:
class Records(Persistent):
__parent__ = None
def __init__(self, parent):
self.__parent__ = parent
self.data = OOBTree()
def __getitem__(self, name):
return self.data.__getitem__(name)
The BTree contains string keys and Record objects as values. Record is:
class Record(Persistent):
__name__ = u
__parent__ = None
field = None
__parent__ is set to records.__parent__, which is going to be the
Registry object.
field is set to a special persistent field class inheriting from
zope.schema's fields, but mixing in persistence. In this case, I have a
list field:
class PersistentField(persistent.Persistent):
Base class for persistent field definitions.
class PersistentCollectionField(PersistentField,
zope.schema._field.AbstractCollection):
class List(PersistentCollectionField, zope.schema.List):
pass
I then have some code like this:
def cloneField(self, field):
clone = field.__class__.__new__(field.__class__)
clone.__dict__.update(field.__dict__)
for name, attr in field.__dict__.items():
if IField.providedBy(attr):
clone.__dict__[name] = self.cloneField(attr)
return clone
This is used like so:
registry = getUtility(IRegistry) # persistent/local
clone = self.cloneField(registry.records[recordName].field)
This seems to work always *except* on the first request immediately
after starting up Zope. In this case, field.__dict__ is {}. If I poke at
it long enough in pdb, it suddenly springs into life and contains values
again.
Can anyone enlighten me as to:
- why this may happen
- how I can ensure it stops happening :-)
You have a ghost, likely: try getting it to unghostify by accessing an
attribute, or calling '_p_activate' (accessing '__dict__' doesn't
provoke the auto-activation).
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAktE7UUACgkQ+gerLs4ltQ68YwCeOywUw4a/c7UXap6uxyIzMCx7
9VcAoII4X56V7LConhNSYCWvFMWMooBD
=XvFD
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTrees and id mapping (was Re: Data.fs size grows non-stop)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Pedro Ferreira wrote: David Glick wrote: Pedro Ferreira wrote: Hello all, So, I've been experimenting with a new structure for our date indexes, and I was considering using an IOBTree that would map integer timestamps to BTree-based sets. As for the sets, I considered two options: Be careful with using timestamps as IOBTree keys. By default this type of BTree uses the system integer, which on many systems is 32 bits. That means that you'll overflow the maximum allowed key for timestamps beyond the year 2038. It won't even give an error; it will just silently overflow and overwrite existing values in the BTree. Yes, I know. That's one of the reasons I'm using mmdd integers instead (the other one being i only need 1 day resolution). Thanks a lot for the tip! An LOBTree uses a guaranteed-to-be-long (64 bits) integer for the keys, if you need that. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksmsuMACgkQ+gerLs4ltQ6mNgCgwBpVRaUli7a7Ij1NQYYHsxif OD0AnjgAGY1TB6GAadx1qSmwW/8w5c1a =ZJRe -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Repozo tests -- not
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Godefroid Chapelle wrote: Godefroid Chapelle wrote: Jim Fulton wrote: snip No one seems to be willing to step up and take responsibility. I guess everyone assumes I will. Jim I'll take the responsibility of transforming the manual test to an automated... Done in svn.zope.org/repos/main/ZODB/branches/test_repozo I added code changes from revisions 105888, 105915 and 105919. Tests pass for me on MacOSX with 2.4, 2.5 and 2.6 Thanks for throwing yourself on that grenade, Godefroid. I have checked in changes to your branch which: - - Use a tempdir for backup / restore. - - Clean out module-scope imports. - - Avoid shellling out to run repozo, but rather use its main(), passing argv. - - Suppress noisy output, unless NOISY_REPOZO_TEST_OUTPUT is in the environment. Remaining cleanup would be to remove the Monte Carlo use of random to generate permutations: the test takes a *long* time to run, because it tries to run 100 times and use random.random to get permutations of the three choices run: these should be six separate tests, likely. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksWuYsACgkQ+gerLs4ltQ7LYwCg0AK9vHca71u7q0maPM86hY2l yzkAoIzgiGiEZLhcuG7vvTnHtTeuDbK2 =QKtA -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Fri, Nov 20, 2009 at 5:49 AM, Chris Withers ch...@simplistix.co.uk wrote: Hi Jim, It would be great if the two recent fixes I committed could make it into a 3.9.4 release some time soon... What can I do to help this happen? Assuming I do anything necessary, when could you do this release? (or give Christian permission to do it :-) ) I need to review the changes before the release. I'll probably reject the repozo change without an automated test. - -sys.maxint. In what possible universe are we better off not being able to run repozo under Python 2.6? That fix is trivial to inspect for correctness, changes no behavior, and is like dozens of other Python 2.6 compatiblitity changes made in our codebase. Repozo is *the* documented way of backing up filestorages: with Zope 2.12 and later now running on Python 2.6, we should just tell people Sorry, you can't do backups? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksGw0cACgkQ+gerLs4ltQ4uHwCgrxg3tCLKKJahyRCgE70AgW1Z wt8An01/eLljzLCtcSDlVd5vfe8jgAhl =DVF7 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Tres Seaver wrote:
Jim Fulton wrote:
On Fri, Nov 20, 2009 at 5:49 AM, Chris Withers ch...@simplistix.co.uk
wrote:
Hi Jim,
It would be great if the two recent fixes I committed could make it into a
3.9.4 release some time soon...
What can I do to help this happen?
Assuming I do anything necessary, when could you do this release?
(or give Christian permission to do it :-) )
I need to review the changes before the release. I'll probably reject the
repozo change without an automated test.
-sys.maxint. In what possible universe are we better off not being able
to run repozo under Python 2.6? That fix is trivial to inspect for
correctness, changes no behavior, and is like dozens of other Python 2.6
compatiblitity changes made in our codebase.
Repozo is *the* documented way of backing up filestorages: with Zope
2.12 and later now running on Python 2.6, we should just tell people
Sorry, you can't do backups?
I just checked in a stupid test to ensure that repozo can be imported.
- --- src/ZODB/scripts/tests.py (revision 105913)
+++ src/ZODB/scripts/tests.py (working copy)
@@ -26,6 +26,11 @@
(re.compile('hash=[0-9a-f]{40}'),
'hash=b16422d09fabdb45d4e4325e4b42d7d6f021d3c3')])
+class RepozoTests(unittest.TestCase):
+
+def test_importability(self):
+from ZODB.scripts import repozo
+
def test_suite():
return unittest.TestSuite((
doctest.DocFileSuite(
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAksGyKkACgkQ+gerLs4ltQ5NDACfXI8srJ2V1K/g3iwB8OOtv4Qb
/8kAoJ6ELmc1vQzePnh4/w71Xw/TJy8O
=6+vL
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.9.4 release?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jim Fulton wrote:
On Fri, Nov 20, 2009 at 11:49 AM, Tres Seaver tsea...@palladion.com wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Tres Seaver wrote:
Jim Fulton wrote:
On Fri, Nov 20, 2009 at 5:49 AM, Chris Withers ch...@simplistix.co.uk
wrote:
Hi Jim,
It would be great if the two recent fixes I committed could make it into a
3.9.4 release some time soon...
What can I do to help this happen?
Assuming I do anything necessary, when could you do this release?
(or give Christian permission to do it :-) )
I need to review the changes before the release. I'll probably reject the
repozo change without an automated test.
-sys.maxint. In what possible universe are we better off not being able
to run repozo under Python 2.6?
You can run repozo under Python 2.6 now. You'll get a deprecation warning.
So we have an annoyance rather than a catastrophe: that helps some. It
will still break on 2.7 without the trivial fix.
That fix is trivial to inspect for
correctness, changes no behavior, and is like dozens of other Python 2.6
compatiblitity changes made in our codebase.
Repozo is *the* documented way of backing up filestorages:
Where is that?
http://wiki.zope.org/ZODB/FileStorageBackup
with Zope
2.12 and later now running on Python 2.6, we should just tell people
Sorry, you can't do backups?
No, we should tell them they should use a mostly untested tool to do backups
that doesn't work with blobs. That should inspire confidence.
We can use rsync to back up blobs safely / losslessly without restarting
the ZEO server: we can't do that with filestorages.
Does Zope 2.12 use Blobs?
If configured, sure.
I just checked in a stupid test to ensure that repozo can be imported.
- --- src/ZODB/scripts/tests.py (revision 105913)
+++ src/ZODB/scripts/tests.py (working copy)
@@ -26,6 +26,11 @@
(re.compile('hash=[0-9a-f]{40}'),
'hash=b16422d09fabdb45d4e4325e4b42d7d6f021d3c3')])
+class RepozoTests(unittest.TestCase):
+
+def test_importability(self):
+from ZODB.scripts import repozo
+
I appreciate your dedication to quality.
That test passes under ZODB 3.9.3 with Python 2.6.
It emits the same DeprecationWarning (the bug Withers fixed). I did
have to munge the 'test_suite' dinosaur as well.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAksG4bIACgkQ+gerLs4ltQ7czQCgzAjQX0TUEJKSNo92+uv8UWOw
VmEAoJ9Xo7WRamCQyuwmY7Kacrqk/Krz
=/5ad
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ValueError: corrupted record when calling GetObject
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Izak Burger wrote: Hi all, On one of our relatively highly loaded zope instances we occasionally get this problem. It runs fine for several days and then all of a sudden it breaks with this traceback: * Module Products.ZCatalog.CatalogBrains, line 86, in getObject * Module ZODB.Connection, line 761, in setstate * Module ZODB.Connection, line 819, in _setstate * Module ZODB.serialize, line 604, in setGhostState * Module ZODB.serialize, line 597, in getState * Module ZODB.serialize, line 471, in _persistent_load * Module ZODB.serialize, line 537, in load_oid * Module ZODB.Connection, line 213, in get * Module ZEO.ClientStorage, line 727, in load * Module ZEO.ClientStorage, line 732, in loadEx * Module ZEO.cache, line 189, in load * Module ZEO.cache, line 1010, in access * Module ZEO.cache, line 618, in fromFile A restart fixes the problem. Its running version 3.7.1 of ZEO/ZODB. That isn't the full traceback: we can't see see the error itself. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrvNBUACgkQ+gerLs4ltQ5sZACfR4gOD1A327rVyGBTWRFcAwFr 2WMAn3HrHfaSTrMbxm2DAooblbIIGk/D =czmU -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/trunk/src/ Fixed a couple of blob storage issues:
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Shane Hathaway wrote: Log message for revision 101802: Fixed a couple of blob storage issues: - The lawn layout was being selected by default if the root of the blob directory happened to contain a hidden file or directory such as .svn. Now hidden files and directories are ignored when choosing the default layout. - BlobStorage was not compatible with MVCC storages because the wrappers were being removed by each database connection. There was also a problem with subtransactions. Fixed. Is the subtransaction problem you fixed related to leakage of savepoint files? https://bugs.launchpad.net/zodb/+bug/323067 Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKWQkI+gerLs4ltQ4RAlAjAJwNX8RhG9/vbC7SXFiQkiv2ylxx4wCbBVbM nWAoq7w2M7vABYeL8/icKQ0= =Sld3 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Blob savepoints leaving garbage?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Theune wrote: On Thu, Jun 26, 2008 at 08:32:25PM +0200, Christian Theune wrote: Hi, I think I didn't totally grasp the way that blob savepoints work correctly. At one of our installations we have lots of .spb files left around. Looking at the code I don't see where .spb files get cleaned up again on savepoint aborts. I reviewed the savepoint tests and see that they don't check for cleanup: neither on rollback, nor on abort. On the bushy branch I found that having the TmpStore create directory structures is probably a bad idea. I propose to have it create a directory `savepoints` in the temporary directory and create files called `oidrepr-tidrepr.spb`. I implemented this on the branch and it seems to work well. I find it easier to inspect on the filesystem than having savepoint state being distributed over the whole structure. Did a fix for the leaked saveepoint stuff ever get committed? https://bugs.launchpad.net/zodb/+bug/323067 Can we safely remove anything under tmp/savepoints, assuming that we know there is not a transaction running? Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKV2O4+gerLs4ltQ4RAjo9AJ0W4ShPfPl7SXCKjQNDm7rNc2ZrGQCglbvL N3WVI/IeVhocIWWNGPnmfDw= =h36T -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Removing the unused and untested ZODB.Mount module
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: It looking for the ZODBMountPoint object that Tarek refered to, I saw that there is still a Mount module in the ZODB package. AFAIK, this a fossil that never should have been included in ZODB in the first place. I'm going to remove it from the trunk and from 3.8. A somewhat late follow up :) : I backported today the change from the Zope2 trunk to the 2.10 and 2.11 branches, moving that module from ZODB to the TemporaryStorage product. It should now be easier for folks to experiment with using ZODB 3.9.x in the earlier Z2 versions. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKMUq2+gerLs4ltQ4RAldwAKCyzdB2WTlMTPszM0zGQMuKBNy7qACfdliy grPiT6MlslndOjVOmWx/MLc= =+Z8y -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ross J. Reedstrom wrote: On Tue, Jun 09, 2009 at 06:29:38PM -0400, Jim Fulton wrote: On Jun 9, 2009, at 4:19 PM, Ross J. Reedstrom wrote: And what's the preferred patch-submission format? *I* prefer svn branches. :) Are you a contributor? Otherwise, a launchpad submission, as Tres suggested, is fine. *checks his zope.org account* 2000? Had it really been that long? Wow. I've got an account, but sadly never completed the contributor agreement (or at least, I don't recall if I have) Hmm, perhaps I should fill one out ... Form: http://foundation.zope.org/agreements/ZopeFoundation_Committer_Agreement Directions: http://docs.zope.org/developer/becoming-a-contributor.html Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKMGsa+gerLs4ltQ4RAjTmAJ9IcE2UsvnKSCKf5huTjB10qcN3NgCg3KSq knpo5ZL9qJOXmoEfLASGcmk= =v60H -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Moving temporary storage from each zeo client to zeo server
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Godefroid Chapelle wrote: I found the following https://weblion.psu.edu/trac/weblion/wiki/TemporaryStorageInZeo where Weblion documents how to move temporary storage from each zeo client to zeo server in Zope2. I understand the pro (no need for sticky sessions). What would be the cons ? - - Increased network overhead - - Extra RAM consumption on the storage server Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKLoXn+gerLs4ltQ4RAgjTAKC8IA2ru1tmaVjOmIU0PS3WOfq07wCfdx89 7bEnwfOkgdSvBTbI5oSQWGk= =Y3zZ -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] fixing far-future timestamps (patch)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ross J. Reedstrom wrote: ... And what's the preferred patch-submission format? Upload it as an attachment to a new issue in launchpad: https://bugs.launchpad.net/zodb using the format:: $ svn diff -u Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKLtaf+gerLs4ltQ4RAtgvAKCA7Son+upU5/JqJfpGEVImBgv1pACdEjYP HCO5JuHVToUsujsZJ1UJ4tk= =+FJ9 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Pedro Ferreira wrote: Thanks a lot for your help. In fact, it was a matter of increasing the maximum recursion limit. There's still an unsolved issue, though. Each time we try to recover a backup using repozo, we get a CRC error. Is this normal? Has it happened to anyone? I don't recall anything like that. Can you provide a traceback? I guess we have a very large database, for what is normal in ZODB applications. Not really: I know of clients whose database routinely grow much larger than yours (15 Gb, packing down to 6 Gb, right?) We were wondering if there's any way to optimize the size (and performance) of such a large database, through the removal of unused objects and useless data. We perform packs in a weekly basis, but we're not sure if this is enough, or if there are other ways of lightening up the DB. Any recommendations regarding this point? Without knowing anything about the application: - - Check that it is not holding onto old data inappropriately (e.g., maintaining lots of archival versions of content). - - Check into the catalog / index usage: you may be able to slow the growth by batching updates, especially to text indexes. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKGsvu+gerLs4ltQ4RApw+AKCOHgdPvKFsEXYnGRngrKLkwz5JrwCeNk+9 6RG4zZcUSw881nEWw5qKAEE= =biZk -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miles Waller wrote: Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miles wrote: Hi, I've moved a FileStorage from one (old) machine to another (new) machine, but when I mount it on the new machine I get a lot of time errrors: Traceback (innermost last): Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module App.special_dtml, line 176, in _exec Module DocumentTemplate.DT_Let, line 76, in render Module DocumentTemplate.DT_In, line 703, in renderwob Module DocumentTemplate.DT_With, line 76, in render Module DocumentTemplate.DT_Var, line 214, in render Module App.PersistentExtra, line 43, in bobobase_modification_time Module DateTime.DateTime, line 509, in __init__ Module DateTime.DateTime, line 760, in _parse_args Module DateTime.DateTime, line 437, in safelocaltime TimeError: The time 98040302366.810165 is beyond the range of this Python implementation. I googled around on this problem and didn't find much, but I did find that as a temporary fix, I've put a try/except in App.PersistentExtra, so if it generates an exception, it returns the current date and time. This keeps things working. Out of interst, can anyone tell me why there is a difference? On both machines, python was compiled from source with the same options, so I'm not sure what this is down to. More importantly, what's the most straightforward way to set all the dates to usable values? From my POV, it's fine to set them all to the current date, but I don't know how to go about doing this exhaustively. Thanks for your help, For each machine: what version of Python and Zope? what OS? Old machine: Python 2.3.5, Zope 2.8.6, OS is Red Hat 8.0 (!) New machine: Python 2.4.3, Zope 2.9.8, OS is Ubuntu 7.10 I had discounted python/zope version differences as other databases had upgraded fine. (CC'ing the newsgroup) Zope is defending against a change in Python's time module, which in turn is due to a change in the platform's time_t: $ bin/python2.3 Python 2.3.7 (#1, Aug 20 2008, 12:39:04) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type help, copyright, credits or license for more information. int(98040302366.81016) 98040302366L from time import gmtime, localtime gmtime(int(98040302366.81016)) (1901, 12, 13, 20, 45, 52, 4, 347, 0) localtime(int(98040302366.81016)) (1901, 12, 13, 15, 45, 52, 4, 347, 0) $ bin/python2.4 Python 2.4.4 (#2, Nov 14 2007, 13:58:47) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 Type help, copyright, credits or license for more information. int(98040302366.81016) 98040302366L from time import gmtime, localtime gmtime(int(98040302366.81016)) Traceback (most recent call last): File stdin, line 1, in ? ValueError: timestamp out of range for platform time_t localtime(int(98040302366.81016)) Traceback (most recent call last): File stdin, line 1, in ? ValueError: timestamp out of range for platform time_t As you can see, the Python 2.3 version has a wraparound problem. I have no idea how you got such an odd value stored for your '_p_mtime', unless you have been using Guido's time machine to run your appserver in 2039. ;) Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJOAcs+gerLs4ltQ4RAooLAKCbzF9wBkwhtnhISmIGqIt22y7+rACfThhx IyjRAFhHuPE+LAVSj4Ct3Mk= =iq+a -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Reliability of 'repozo --quick' option
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Does anybody have evidence or belief that the probabalistic part of the '--quick' optoin (as of ZODB 3.2.8, if it matters) is likely to guess wrong on a setup where incremental backups are run frequently? The installation in question has a moderately large filestorage (40 Gb or so) and would like to put the backup target on a SAN, but the cost to figure out whether to do an incremental or not is higher than doing the full backup, due to the extra I/O overhead of the standard, slow method. Before I hack the backup script up to do the incrementals against a local directlry, I'd like to know that the '-Q' option would or wouldn't be a viable choice. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI+5xt+gerLs4ltQ4RAsXxAJ9obJPdK2SnTvF2rfY+9Q+VI1belQCgkkwF TnfpBF5B+N1ucUxcqzDEikM= =kFzL -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Object uniqueness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Markus Läll wrote: I think I know now -- objects compare identical and equal only if they are in the same container object. (Check the code below) I don't mean to be negative, but why can't ZODB be like a big RAM? Like when i store an object to multiple places on the same DB, then it would actually be the same in each place. (Like if you comment out the Commit, close and reload part of the below code, all the comparisons evaluate to True) If your classes derive from persistence.Persistent, then their instances' identities will be preserved across serializations. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI5kb7+gerLs4ltQ4RAq3yAJ9USY6EX+cssrxT0ZOOlS6EG+OREwCfYn5g MqQBgyaT2JnWYe18jJSI/pE= =Z0o3 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN: ZODB/branches/3.8/src/ZODB/ Removed the feature to try to save the index periodically because:
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: Log message for revision 91235: Removed the feature to try to save the index periodically because: 1. The saving was performed in (tpc)_finish. It is important that this method do as little as possible because it cannot fail! 2. The algorithm for deciding how often to save was broken. For large databases, for which saving periodically is desireable, the period was set so high that the index was effectively never saved. It might be nice to save periodically, but doing so is tricky, since we really don't want to do it during commit. Until we figure out how to do this right, it is better not to try. In the mean time, we save on close and on pack, which is proably often enough in most cases. Perhaps we could extend the zeopack RPC mechanism with a special value for the pack interval: if passed, the server would just do the index save, and skip the real packing. People could then tune the interval by scripting. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI0m78+gerLs4ltQ4RAqjxAJwJXrWyPtY3ze0N0eSxu9Z9j3SxLQCgwXBU 7uxjYZeUlc/uOlmpO39okNs= =ZB86 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Reimplementing Pickle Cache in Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dieter Maurer wrote: Tres Seaver wrote at 2008-9-17 15:06 -0400: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dieter Maurer wrote: Tres Seaver wrote at 2008-9-12 06:35 -0400: ... Reimplementing Pickle Cache in Python = ... from zope.interface import Attribute from zope.interface import Interface class IPickleCache(Interface): API of the cache for a ZODB connection. ... Which method moves an object to the front of the ring? Or do you use an inline expansion for speed reasons? Thanks for catching my error. In the current implementation, the C version of the Persistent base class calls a static C helper function inside its 'accessed' method, which is called from the various attribute getter and setters. My implementation of the ring in Python needs to grow another method which takes the place of that helper. Something like: class IPickleCache(Interface): def mru(oid): Move the element corresonding to 'oid' to the head. The implementation would look something like: class PickleCache(object): ... def mru(self, oid): node = self.ring.next while node is not self.ring and node.object._p_oid != oid: node = node.next The while worries me a bit: mru is called for any attribute access and must therefore strive for efficiency. I think we could relax the any attribute access bit and still win (in fact, that seems absurd on the face of it). I don't know what the secret would be: maybe we have to track an mtime of some kind, and bump the 'mru()' method only when the mtime changes (no more than once per transaction, even for reads, seems sane). In fact, one could argue that attribute access is exactly *wrong*: the right time to hit move the item to the head of the list is when the object is fetched from the connection. Another problem is thread safety. The C implementation could rely on the GIL being held. I fear that a Python implementation would need to use explicit locking. That might be problematic for the general case, although we never share a connection across threads in Zope. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI0pRR+gerLs4ltQ4RAqnDAJ91QU49ymb2ypG/VgZkDBX0WHLTlACfU+aX mJ5as7n5q6kCFtM+HfnX0Q4= =EvUK -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: Reimplementing Pickle Cache in Python
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Dieter Maurer wrote:
Tres Seaver wrote at 2008-9-12 06:35 -0400:
...
Reimplementing Pickle Cache in Python
=
...
from zope.interface import Attribute
from zope.interface import Interface
class IPickleCache(Interface):
API of the cache for a ZODB connection.
...
Which method moves an object to the front of the ring?
Or do you use an inline expansion for speed reasons?
Thanks for catching my error. In the current implementation, the C
version of the Persistent base class calls a static C helper function
inside its 'accessed' method, which is called from the various attribute
getter and setters.
My implementation of the ring in Python needs to grow another method
which takes the place of that helper. Something like:
class IPickleCache(Interface):
def mru(oid):
Move the element corresonding to 'oid' to the head.
The implementation would look something like:
class PickleCache(object):
...
def mru(self, oid):
node = self.ring.next
while node is not self.ring and node.object._p_oid != oid:
node = node.next
if node is self.ring:
raise KeyError('Unknown oid: %s' % oid)
# remove from old location
node.prev.next, node.next.pref = node.next, node.prev
# splice into new
self.ring.prev, node.next = node, self.ring
That method would then be called by the corresponding 'acceseed' (or is
it '_p_accessed'?) method in the new Persistent class.
Tres.
- --
===
Tres Seaver +1 540-429-0999 [EMAIL PROTECTED]
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFI0VU/+gerLs4ltQ4RAtb6AJ98MBvPQ+4DTDs6TDYNfaXPhLVJZwCfd6bd
prN6YvDRBhc1lbwwfFnsR/g=
=UOSG
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Backing up Data.fs and blob directory
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Christian Theune wrote: Hi Laurence, On Wed, 2008-09-03 at 08:06 -0700, Laurence Rowe wrote: Backing up a ZODB has always been fairly easy in the past, but with the introduction of blobs things have got a little more complex. How should I create a consistent backup of my Data.fs and blob directory? My inital guess would be to take a copy of the Data.fs, then take a copy of the blob directory to ensure I have all blobs referenced in the Data.fs. Would I be able to restore from such a backup safely? (it may contain blobs from transactions that were newer than the backed up Data.fs). This should be safe because committed blobs are immutable and any dangling blobfiles would not interfere with the creation of blobs from new transactions in the restored zodb, as transaction ids would not overlap. I would be greatful if anyone could point out holes in my reasoning or has experience of this. Snapshotting a blob directory after taking a copy of your Data.fs should be safe, as long as you don't pack in between. Note that at the design stage we imagined that blob directories might become really large making backups unfeasable. For those situations we handwaved a very reliable storage for this directory, like a self-contained SAN/NAS solution that keeps your data safe. Assuming we can avoid the race condition induced by packing during the backup (see below), it should be possible to write a script which combines of 'repozo' and 'rsync' in such a way as to get a pure copy of the blob directory which corresponds to the repozo dataset. Somethiing like: #!/bin/sh touch .repozo_start /path/to/repozo -B -f var/Data.fs -r backups/ find /path/to/blobs ! -newer .repozo_start |\ rsync -av --include-from=- /path/to/blobs/ backups/blobs/ rm .repozo_start Packing *during* backup creates problems for repozo + blobs, because a pack may cause old blob files to be unlined. Furthermore, it is not a use case I think we should support. However, I don't think that FileStorage supports the idea of a pack lock which would be acquirable by repozo, so I don't know how to prevent the race. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIwATJ+gerLs4ltQ4RAm0WAKC/jDUPqBnMTpwkDpBX0mKidGTxvwCghjnM QYnz9dRolzOdvZX2t9fxM3k= =/h4K -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] test.py in ZODB3-3.8.1b7.tar.gz
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 richrad miller wrote: I am new to zodb. I have downloaded zodb3-3.8.1b7 and all seems to run well. I was wanting to run the tests as advertised in the README.txt. I am unable to find a test.py. Am I being stupid? Nope. The test driver is geerated by the buildout, so you need to do the boostrap + buildout process first. At that point, the tests are runnable from 'bin/test'. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIvqy7+gerLs4ltQ4RAsYkAJ0W5kcbNq1fqyWXuhP0fPTLniKpQwCfRSG6 IbN4abpQ83VRZV4BGPTnQ/g= =l8vP -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree pickle size
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Roché Compaan wrote: On Fri, 2008-08-22 at 16:37 -0300, Sidnei da Silva wrote: On Fri, Aug 22, 2008 at 9:49 AM, Roché Compaan [EMAIL PROTECTED] wrot Transaction detail for txn #00099 (first document): Txn id,Classname,Object count,Size (bytes) #00099,BTrees._IIBTree.IIBTree,3,286 #00099,OFS.Folder.Folder,1,55 #00099,BTrees._IOBTree.IOBucket,9,4572 #00099,BTrees._OIBTree.OIBucket,5,2964 #00099,BTrees._IOBTree.IOBTree,39,17552 #00099,BTrees.Length.Length,27,768 #00099,Persistence.mapping.PersistentMapping,2,846 #00099,Products.ATContentTypes.content.document.ATDocument,1,1544 #00099,BTrees._OOBTree.OOBTree,20,3986 #00099,BTrees._IIBTree.IISet,3,184 #00099,BTrees._OIBTree.OIBTree,9,1404 #00099,Products.Archetypes.BaseUnit.BaseUnit,3,767 #00099,BTrees._OOBTree.OOBucket,2,3286 #00099,BTrees._IIBTree.IITreeSet,55,3905 ?Transaction detail for txn #10099 (last document): Txn id,Classname,Object count,Size (bytes) #10099,BTrees._IIBTree.IIBTree,8,2517 #10099,OFS.Folder.Folder,1,55 #10099,BTrees._IOBTree.IOBucket,57,81564 #10099,BTrees._OIBTree.OIBucket,13,9872 #10099,BTrees._IIBTree.IIBucket,29,20024 #10099,BTrees._IOBTree.IOBTree,1,85 #10099,Persistence.mapping.PersistentMapping,2,846 #10099,BTrees.Length.Length,22,655 #10099,Products.ATContentTypes.content.document.ATDocument,1,1544 #10099,BTrees._OOBTree.OOBTree,6,30455 #10099,BTrees._IIBTree.IISet,65,182708 #10099,Products.Archetypes.BaseUnit.BaseUnit,3,767 #10099,BTrees._OOBTree.OOBucket,16,8088 #10099,BTrees._IIBTree.IITreeSet,2,122 It's pretty clear that the difference here is the IISet(65 vs 3) and the IOBucket(57 vs 9). The rest looks pretty much stable. Now, if I understand correctly that means the last document caused 57 IOBuckets to be modified, but not necessarily created. Right. But even looking at the very first transaction the indexing overhead is visible: 3 Kbytes of data related to the document (ATDoc, BaseUnit, PersistentMapping) is only a fraction of the total transaction size of 40 Kbytes. I recall a pre-Zope (for me, 10 years ago) rule of thumb that text indexing imposed an order of magnitude of overhead on the actual corpus, with improvements possible only via batching or post-processing / compresstion (incremental indexing is worst-case). Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIryVR+gerLs4ltQ4RAv7CAKC68bT3zmp5P1xOpxCX+TpoVg/qJACcC1rv 5oQeHxjFc3iCkJz8o09awP0= =wYKj -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: ZODB + ZEO performances problem
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thierry Florac wrote: I'm currently trying to migrate an old Zope-2.6.1 application to Zope-2.9.9. The new application platform is a Sun Sparc Solaris 10 server (dual processors). Application setup includes a ZEO server managing two databases (a main one for common data and a catalog to store catalog), and two Zope clients (one as main frontend and another one for uncommon administration tasks, both with standard threads count of 4). Database sizes (when packed) are of 280 and 400 MB ; Zope clients caches are of 300 and 500 MB, for 4 and 8 objects. All data are stored on two mirrored (via RAID 1) Hitachi SAN with quite good writing speed. Because you need fast *seek* speed, too, I would recommend trying your app with the ZODB on local disk, first. In fact, I never recommend putting the main ZODB on a non-local disk, for just this reason. The usage pattern of FileStorage (one big file, writing only at the end, but with lots of random-access reads throughout) is nearly worst-case for such deployments. I have one client whose setup is similar (Linux, but using a big iron SAN), where the first phase of incremental repozo backup (the part checking whether incremental is possible) takes longer by a factor of two than just doing a full backup. Putting it on local disk both makes writes go faster and makes incremental backup feasible. If you must stay with the SAN, I would investigate whether DirectoryStorage, which uses lots of litte files, is a better fit. I actually have two problems with this configuration : - storing an object into ZODB is VERY slow ; it can take more than 30 seconds when the object wasn't saved recently, event under light load. I actually suspect that it's the catalog full-text search index which is long to load, update and store. The old application under Zope-2.6.1 didn't have this problem (but also didn't use ZEO !). Is there a specific way to handle catalogs with ZEO, should I avoid this kind of configuration ? You can change the application to use QueueCatalog, which defers writes to a configurable set of indexes (typically the text index(es)) for later processing. I often run that processing via a cron job or other long-running, non-server process (running as a ZEO client). - sometimes (but most often under quite heavy load) the Zope clients completely hang and stop responding, without any access or error log, and without any disk or CPU activity. I suppose it's a kind of deadlock, how can I detect the source of the problem ? Many thanks for any help or advise, as the resolution of these problems is quite urgent... As Alan said, the DeadlockDebugger product is designed to help diagnose such issues. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIfp7e+gerLs4ltQ4RAgmfAJ9K9TtMfLc3i0hndANvvrY/PaA9qwCgrof8 YAGIUADOgpkIJvb2A7c5JJM= =76B6 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Blob directory structure scalability limits
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Martijn Faassen wrote: Hi there, On Thu, Jun 19, 2008 at 1:38 PM, Christian Theune [EMAIL PROTECTED] wrote: [snip] We propose to keep both implementations around and allow to select which one to use. We would extend the FileSystemHelper to abstract the two strategies. Could the default be to choose the more scalable one, or would this lead to backwards compatibility issues? Perhaps the class-level default would be the old one, but the constructor-argument-default the new? E.g.: class BlobStorage: strategy = 'naive' def __init__(self, strategy='scalable'): self.strategy = strategy Hmm, but OTOH, this is not likely a case where we have to worry about old persistent data ;). I guess old config files need to continue to work as expected, which means that the directory itself needs some kind of metadata file which indicates the strategy: in that case, the absence of the file would indicate the need for the BBB version. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIWl4b+gerLs4ltQ4RAvsLAJ9bcsMk9KcZ/qnWE2NWTmaPWMlK3wCeLg5Q 8zPnqg89CLeF7AAvfM91egA= =gGxz -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: ZODB Benchmarks
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris Withers wrote: Roché Compaan wrote: Not yet, they are very time consuming. I plan to do the same tests over ZEO next to determine what overhead ZEO introduces. Remember to try introducing more app servers and see where the bottleneck comes ;-) Am I right in thinking the storage servers is still essentially single threaded? Yes, but they are normally network / disk bound, rather than CPU bound, which makes that less of an issue. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH4xqu+gerLs4ltQ4RApy5AJ9D3nzvv9329p5hdeZZSUj+smbPrQCfe5nd 8tInjsl71RrpSVOD7DVADYE= =129a -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Zeo Server as a single point of failure
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David Pratt wrote: Hi Alan. I'm keeping the thread on the list since you raised the issue here. There are several repositories of zope related code that exist outside of svn.zope.org. I have made the decision to host DirectoryStorage at sourceforge on the basis that I have committed to maintaining and contributing to the project. I volunteered to maintain the project since I saw it falling into disrepair following two years without a single commit to the code base. I have become the maintainer on the basis of an agreement with Tobi back in January. This agreement permits me to relicense the code under my choice of a ZPL or MIT license. I cannot work with sources in zope repository since it would require a contributor agreement with Zope Corp. I am unable to enter into this agreement for the forseeable future. The zif collective (zif.sourceforge.net) is a way that I may contribute back to the zope community without the requirement of the agreement. You don't identify the problem(s) which prevent you from entering into that agreement: we might be able to clarify or help resolve them if you did so. Anyone interested in DirectoryStorage is encouraged to participate in its development since it is an open project that will be getting a more liberal license. There will be something more to build upon once the egg repackaging is released. It will be released under zif namespaced package. Shane and I have a relationship since I was also a project admin for PGStorage when it was located on sourceforge. There is nothing that would prohibit our communication or collaboration on any future development of DirectoryStorage. HTH. What does Shane have to do with DirectoryStorage? That product was originated by Toby Dickinson, based on a prototype by Petru Paler, with contributions from several other folks, per: http://dirstorage.sourceforge.net/ and http://dirstorage.cvs.sourceforge.net/dirstorage/DirectoryStorage/doc/HISTORY.txt?revision=1.2 Relicensing it will require consent from Toby and the other copyright holders: I'm assuming that the arrangment you discuss above fits the bill there. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFH2Bt/+gerLs4ltQ4RAkdvAKCrrZ9GH11Sxh+GHxI/yE349ErOPgCfQfUp v8wLBoKQ547ZgTdFfnj/dSU= =6Y6e -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
