Re: [ZODB-Dev] [zopefoundation/ZODB] 49919d: test for POSKeyError during transaction commit
On Tue, Feb 04, 2014 at 12:44:09PM -0500, Tres Seaver wrote: On 02/04/2014 06:28 AM, Godefroid Chapelle wrote: Le 03/02/14 20:53, Tres Seaver a écrit : I wish you hadn't pushed that -- some of these changes are definitely inappropriate on the 3.10 branch (adding an Acquisition dependency is definitely wrong). Acquisition is added as a test dependency. Any hint how to replicate the bug without acquisition is welcome. Define a subclass of Persistent which emulates what Acquisition does, e.g.: from persistent import Persistent class Foo(Persistent): @property def _p_jar(self): # or whatever attribute trggers return object() What if full replication requires a C extension module? (I hope that's not true and that it is possible to reproduce the bug using some fakes, but I haven't spent the time investigating this.) Which other change is inappropriate ? Adding MANIFEST.in on a release branch seems wrong to me (I don't like them anyway, and we *definitely* don't want to encourage instsall-from-a-github-generated-tarball on a release branch). That's like objecting if someone adds a .gitignore to a release branch. Or a .travis.yml. It's not code, it's metadata. (I never liked setuptool's magic let me query git to see what source files you have, but not by default, oh no, instead let's assume everybody has installed the non-standard plugin into their system Pythons and then let's silently produce broken tarballs if they haven't, because obviously implicit is better than explicit, and when there's temptation the right thing is to guess behavior anyway, and we *definitely* don't want broken sdists on PyPI.) Marius Gedminas -- QOTD: A child of 5 could understand this! Fetch me a child of 5. signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] POSKeyError when transaction.commit after savepoint.rollback and connection.readCurrent
On Mon, Feb 03, 2014 at 02:50:42PM +0100, Godefroid Chapelle wrote: It needed time and obstinacy to find how to write a test that triggers the same POSKeyError during commit as triggered by some Plone 3 to 4 migrations. For more details about the error, see http://rpatterson.net/blog/poskeyerror-during-commit The test is now pushed to the 3.10 branch on github : https://github.com/zopefoundation/ZODB/commit/49919d31cfa73410cede0af969d80527a9fd852c The fix is pushed as well. https://github.com/zopefoundation/ZODB/commit/9ce470f70acc9655aaf915e5a74678b43f6312da ... - As the test above depends on Acquisition (which is not ported to Python 3), what should be the strategy to include my work in master ? Pull the fix only ? Pull the fix and do some magic with the test so that it runs only in Python 2 ? Other suggestion ? Could you tell us why that test depends on Acquisition? Is it possible to replicate the bug using pure Python code? Marius Gedminas -- Q: A good ninja is hard to find. A: Well, yes, actually, being hard to find is the whole point with them. -- http://tinyurl.com/5tnrwr signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] POSKeyError when transaction.commit after savepoint.rollback and connection.readCurrent
On Mon, Feb 03, 2014 at 03:33:13PM +0100, Godefroid Chapelle wrote: Le 03/02/14 15:12, Marius Gedminas a écrit : Could you tell us why that test depends on Acquisition? Is it possible to replicate the bug using pure Python code? I have not been able to replicate the bug with pure Python code. Reason why it took me so much time. IIRC, the issue is that on newly created objects that inherit from Acquisition.Implicit, the _p_jar attribute has a value (I guess it is acquired). Without acquisition, because a newly created BTree._p_jar does not have any value before transaction is committed, any call to readCurrent is then discarded. Ah, I thought it might be caused by the acquisition of _p_oid, but then decided that made no sense. (Actually I thought Acquisition would skip _p_* or _v_* attributes, and I'm surprised to hear that it doesn't.) I understand this gives the feeling that the fix should be in Acquisition rather than in ZODB. I have tried to make a fix for the Acquisition code but was unable to understand enough. Nevertheless, my understanding of the readCurrent API makes me think that it is actually useless to keep reference to newly created objects (iow in same transaction). And that the fix I propose makes sense anyway. I'm not disputing that. (This is way beyond my familiarity with ZODB internals.) Marius Gedminas -- question = (to) ? be : !be; -- Shakespeare signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Thu, Aug 22, 2013 at 03:40:41PM -0600, Sean Upton wrote:
Caveat to this below.
On Mon, Jul 15, 2013 at 7:54 AM, Marius Gedminas mar...@gedmin.as wrote:
On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote:
We need to move a considerable number of persistent objects (~40K) from
one module to another, and we were wondering what is the best strategy
to follow.
...
Has anyone ever done anything like this? Which approach have you
followed? Any suggestions?
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
That is sufficient to make it work. If you also want to eradicate all
references to the old module name from your ZODB (e.g. because you'd
like to remove the BBB import), then proceed to step 2:
2. Write a script that loads every instance of this class and does a
obj._p_activate() # actually not sure this is required, but won't hurt
obj._p_changed = True
and then commit the transaction. Do the commit multiple times, after
each batch of several hundred objects, to avoid excessive memory usage.
Also be sure to handle conflict errors and retry that batch if you're
running this script on the live system. Finding all instances is left
as an exercise for the reader (sometimes findObjectsProviding() helps,
if you use nested containers everywhere; sometimes application-specific
logic works best; sometimes you end up having to use ZODB iterators to
loop through every single object in the DB -- I believe zodbupdate does
that.)
This works only partially, AFAICT. It will update the stored class name of
a persistent object. It will not seek out and change the class name of the
object in a reference. Your step 3 (removing the BBB code) can break
things (broken objects) as a result of this. Fixing persistent objects is
half the battle if you get broken references to them.
Ouch, you're absolutely right.
You can verify that this is incomplete by creating a PersistentMapping of
some simple objects, perform the trick above, commit, then pack. You will
still have the old BBB classname stored in the database in the references,
unless you do a _p_changed=1 on the mapping containing/referencing the
items as well (then commit).
I am not sure how to walk/iterate all oids for all transactions to get all
possible referencing objects (I assume this is storage-specific, maybe
building a reference map like Lawrence Rowe has done [1]). This might be
necessary to update the referencing objects?
ZODB has an iterator API that allows you to access all the objects.
I think SchoolTool used to have an evolution script that did precisely
this (walk all the objects, set _p_changed to force a write) to solve
this problem.
And then there's https://pypi.python.org/pypi/zodbupdate which sounds
exactly like the tool you want for this.
[1] http://plone.org/documentation/kb/debug-zodb-bloat/inspectZodbUtils.py
Marius Gedminas
--
Killing gnome-session is likely to free substantial amounts of memory, but the
user's gratitude may be surprisingly limited.
-- Jonathan Corbet in a LWN article about the OOM killer
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Tue, Jul 16, 2013 at 08:54:55PM +0200, Pedro Ferreira wrote:
Hello,
Thanks for your answer.
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
So, I assume this is based on the classFactory hook/dictionary solution?
Or am I misunderstanding it?
No, this is based on ZODB's use of pickles. The database stores the
fully-qualified name of the module and class of each object, so if you
make the classes importable using the old module and class name, the
objects can still be loaded.
Marius Gedminas
--
Moore's Law, I need hardly remind a top-notch industry professional like you,
states that as the density of silicon circuitry doubles, the probability of you
not being able to find some sensibly-priced extra memory to fit your old lappy
approaches 1.0.
-- Verity Stob
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Changing namespace - best strategy
On Mon, Jul 15, 2013 at 03:36:21PM +0200, Pedro Ferreira wrote:
We need to move a considerable number of persistent objects (~40K) from
one module to another, and we were wondering what is the best strategy
to follow.
...
Has anyone ever done anything like this? Which approach have you
followed? Any suggestions?
1. Make sure all the classes are still importable from the old location
('from newmodule import MyPersistentSomething # BBB')
That is sufficient to make it work. If you also want to eradicate all
references to the old module name from your ZODB (e.g. because you'd
like to remove the BBB import), then proceed to step 2:
2. Write a script that loads every instance of this class and does a
obj._p_activate() # actually not sure this is required, but won't hurt
obj._p_changed = True
and then commit the transaction. Do the commit multiple times, after
each batch of several hundred objects, to avoid excessive memory usage.
Also be sure to handle conflict errors and retry that batch if you're
running this script on the live system. Finding all instances is left
as an exercise for the reader (sometimes findObjectsProviding() helps,
if you use nested containers everywhere; sometimes application-specific
logic works best; sometimes you end up having to use ZODB iterators to
loop through every single object in the DB -- I believe zodbupdate does
that.)
3. (Optional) Remove the BBB import added in step 1.
HTH,
Marius Gedminas
--
Voodoo Programming: Things programmers do that they know shouldn't work but
they try anyway, and which sometimes actually work, such as recompiling
everything.
-- Karl Lehenbauer
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] BTree data loss bug under Python 3.x
On Fri, May 24, 2013 at 09:31:55PM -0400, Tres Seaver wrote:
On 05/24/2013 01:28 PM, Marius Gedminas wrote:
I was trying to debug a failing ZEO test case
(testConcurrentUpdates), and found out it had nothing to do with
ZEO. There's a bug either in BTrees or in ZODB itself that causes
data corruption under Python 3.x.
Here's a test case:
https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py
Thanks for the detective work! I can confirm the bug, and that it is in
the BTrees C extensions (the tests pass after 'rm
.tox/py32/lib/python3.2/site-packages/BTrees/*.so').
It's the COMPARE macro in _compat.h. This patch fixes the data corruption:
diff --git a/BTrees/_compat.h b/BTrees/_compat.h
index e004d54..4765df1 100644
--- a/BTrees/_compat.h
+++ b/BTrees/_compat.h
@@ -28,8 +28,8 @@
#define TEXT_FORMAT PyUnicode_Format
#define COMPARE(lhs, rhs) \
-PyObject_RichCompareBool((lhs), (rhs), Py_LT) 0 ? -1 : \
-(PyObject_RichCompareBool((lhs), (rhs), Py_EQ) 0 ? 0 : 1)
+(PyObject_RichCompareBool((lhs), (rhs), Py_LT) 0 ? -1 : \
+ (PyObject_RichCompareBool((lhs), (rhs), Py_EQ) 0 ? 0 : 1))
#else
I'm now trying to write a failing unit test. It's a bit difficult to
get access to C macros from Python tests ;-)
COMPARE is used in two places:
BTrees/objectkeymacros.h-#define TEST_KEY_SET_OR(V, KEY, TARGET) \
BTrees/objectkeymacros.h:if ( ( (V) = COMPARE((KEY),(TARGET)) ),
PyErr_Occurred() )
and
BTrees/objectvaluemacros.h:#define TEST_VALUE(VALUE, TARGET)
COMPARE((VALUE),(TARGET))
We can ignore TEST_KEY_SET_OR, as it works fine there.
TEST_VALUE is used to implement BTree.byValue and bucket merging. The
bug I was chasing down was caused by bucket mismerges, I think.
It should be easier to write a test for byValue. In fact there is one,
in test__base.py, but it's not run against real C-implemented BTrees. I
copied it over into test_OOBTree.py and discovered that it fails
unexpectedly:
diff --git a/BTrees/tests/test_OOBTree.py b/BTrees/tests/test_OOBTree.py
index bc046bc..e813f88 100644
--- a/BTrees/tests/test_OOBTree.py
+++ b/BTrees/tests/test_OOBTree.py
@@ -99,9 +99,15 @@ class OOSetPyTest(ExtendedSetTests, unittest.TestCase):
class OOBTreeTest(BTreeTests, unittest.TestCase):
-def _makeOne(self):
+def _makeOne(self, *args):
from BTrees.OOBTree import OOBTree
-return OOBTree()
+return OOBTree(*args)
+
+def test_byValue(self):
+ITEMS = [(y, x) for x, y in enumerate('abcdefghijklmnopqrstuvwxyz')]
+tree = self._makeOne(ITEMS)
+self.assertEqual(list(tree.byValue(22)),
+ [(y, x) for x, y in ITEMS[22:]])
def testRejectDefaultComparison(self):
# Check that passing int keys w default comparison fails.
@@ -146,9 +152,9 @@ class OOBTreePyTest(OOBTreeTest):
# default 'object' comparison semantics.
#class OOBTreePyTest(BTreeTests, unittest.TestCase):
-def _makeOne(self):
+def _makeOne(self, *args):
from BTrees.OOBTree import OOBTreePy
-return OOBTreePy()
+return OOBTreePy(*args)
The failure is
AssertionError: Lists differ: [(25, 'z'), (24, 'y'), (23, 'x... != [(22, 'w'),
(23, 'x'), (24, 'y...
First differing element 0:
(25, 'z')
(22, 'w')
- [(25, 'z'), (24, 'y'), (23, 'x'), (22, 'w')]
+ [(22, 'w'), (23, 'x'), (24, 'y'), (25, 'z')]
so the C-implemented BTrees return values in descending order, while the base
Python implementation returns them in ascending order. This happens on Python
2.7 as well, so it's not related to the COMPARE macro bug.
Does anybody know what the API of BTree.byValue() is supposed to be?
The interface just says Return a sequence of (value, key) pairs, sorted
by value, without clarifying if it means an ascending or a descending
sort.
Marius Gedminas
--
If vegetarians eat vegetables, what do humanitarians eat?
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] BTree data loss bug under Python 3.x
I was trying to debug a failing ZEO test case (testConcurrentUpdates),
and found out it had nothing to do with ZEO. There's a bug either in
BTrees or in ZODB itself that causes data corruption under Python 3.x.
Here's a test case:
https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py
It initializes an OOBTree to {0:0, 1:0, 2:0}, then launches two
threads that attempt just one write each:
tree[1] = 1 # thread 1
tree[2] = 2 # thread 2
All transactions are committed successfully and the end result _sometimes_
is {0:0, 1:0, 2:2} instead of the expected {0:0, 1:1, 2:2}.
There's also a larger test case (zodbfail.py) that modifies more than
one item in a thread. That one fails pretty reliably.
This only happens under Python 3.x.
To reproduce::
git clone git://gist.github.com/5644876.git zodbfail
cd zodbfail
detox
Marius Gedminas
--
Frameworks ought to gracefully fade away as you replace them, bit by bit, with
domain-specific code
-- Jacob Kaplan-Moss
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZEO test failures
On Tue, Mar 19, 2013 at 02:04:05PM +0200, Albertas Agejevas wrote: ZEO master has two failing tests (client_has_newer_data_than_server Tres tried to fix it, and then I tried to fix it. Winbot still finds fault with our fixes :-( and zeo-fan-out.test). This is a bit non-deterministic. There's another similar nondeterministic failure in new_addr.test. Looks like master is not tested by any automated systems. We now have http://winbot.zope.org/builders/ZEO_py_265_32 and https://travis-ci.org/zopefoundation/ZEO Also, I've mostly made the py3 branch to pass tests on Python 3.3, there are the two failures that also appear on master, I've also seen failures in checkReadOnlyFallbackReadOnlyServer, checkReconnectSwitch, checkQuickVerificationWith2Clients, and checkExceptionLogsAtError on Python 3.x. These don't appear to be reproducible in isolation. I'm not sure if they're nondeterministic failures, or if some other test leaves unclean global state that trips up later tests. and one that appears a genuine problem (checkConcurrentLargeUpdates). In this test there are three threads showering a BTree with updates with commits in batches of ~50. Looks like some of the commits disappear. It looks to me like a bug in conflict resolution. I looked at the test DB with zodbbrowser: its not that commits disappear but rather new commits (made by a different thread) revert some OOBTree keys to older values. Scary. Could someone more familiar with ZEO look into these failures? I'm not familiar with ZEO. Marius Gedminas -- Nobody will ever need more than 640k RAM! -- Bill Gates, 1981 Windows 95 needs at least 8 MB RAM. -- Bill Gates, 1996 Nobody will ever need Windows 95. -- logical conclusion signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Towards ZODB on Python 3
stuff on Python 2 or Python 3, you get to deal with bytes becoming unicode when you unpickle. These kinds of guessing games are inevitable when you're migrating pickles from Python 2 to Python 3, but do we want to make them mandatory for day-to-day operation? Perhaps we ought to drop our original goal (3) and require an explicit one-time possibly-lossy conversion process for goal (2), then use pickle protocol 3 on Python 3 and have short pickles, perfect roundtripping of bytestrings? Then there's ZEO, which uses pickles for both payloads _and_ for marshalling in its RPC layer. That's also fun, but I think we can at least declare that ZEO server and client must be on the same Python version, perhaps by bumping the protocol version. So, this is where things stand right now. Plus a few relatively minor matters like adding missing noload() tests to zodbpickle and making zodbpickle work on Python 3.2 [9] [9] https://mail.zope.org/pipermail/checkins/2013-March/065813.html Other than that, the ZODB py3 branch works on Python 3.3 [10]. As long as you're prepared to deal with bytestrings magically transforming into unicodes. [10] Stephan reported running an actual small demo application with it. Where do we go from here? Marius Gedminas -- Basically, what Ajax means is Javascript now works. -- Paul Graham signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB Py3k porting status
On Sat, Mar 02, 2013 at 08:31:37PM -0500, Tres Seaver wrote: Thanks to Marius and Stephan's efforts (with a little help from me), the 'py3' branch of ZODB now passes all unit tests, with most of the many-many ResourceWarnings now suppressed as well (we are down to the last two unclosed blob files). We are? Crud, I thought I got them all. Today, I worked on making the branch work with Python 3.2. In order to support 3.2, I needed a 3.2-compatible verison of 'zodbpickle', as well as ZConfig. The ZConfig changes were trivial (avoiding u'' literals): I pushed out a 3.0.3 release with those changes. For 'zodbpickle', I created a second fork of '_pickle.c' (from the Python 3.2.3 release), and select between the two variants at build time based on the version of Python. I couldn't make a release there (Marius is the sole PyPI owner / maintainer), That was an oversight. You're now an owner on PyPI, and so is Stephan, in case he needs to run his magic script to grant everyone rights. (I hear PyCon is approaching. Any chance of a PyPI sprint to add groups/organizations to the authorization model?) but was able to install it in my ZODB checkout's '.tox/py32' virtualenv for testing. An update for further plans before the ZODB py3 branch can be considered to be merge-ready: * we want to use noload() for garbage collection etc: 1. Python 3's pickle doesn't support it 2. zodbpickle's noload() can't handle the way Python 3 pickles OIDs (as a function call to _codecs.encode(u\x00\x00\x00..., latin-1) instead of a SHORT_BINSTRING b\x00\x00\x00...) 3. instead of changing noload() to interpret function calls we want to change zodbpickle's Pickler to pickle binary strings using SHORT_BINSTRING, i.e. the way Python 2.x is pickling them. * there are a few cases scattered through the code base that check sys.version to see whether they need to assign to inst_persistent_id or just persistent_id on the Pickler. I'd like to move that check into _compat.py. * there were zero failing tests before we switched to zodbpickle, which indicates missing tests: pickles produces on Python 2 most likely cannot be loaded on Python 3 (that was one of the main reasons for zodbpickle's existence). Marius Gedminas -- You know, you never truly appreciate how handy living next to a creek is until you are on fire. -- James Nicoll, rec.arts.sf.written, 1/25/05 signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] what's the latest on zodb/zeo+memcached?
On Thu, Jan 17, 2013 at 12:31:52PM -0500, Claudiu Saftoiu wrote:
I wrote the following code to preload the indices:
def preload_index_btree(index_name, index_type, btree):
print ((Preloading '%s' %s index btree...)) % (index_name,
index_type)
start = last_print = time.time()
for i, item in enumerate(btree.items()):
item
That's a no-op: you might as well just write 'pass' here.
If you want to load the btree item into cache, you need to do
item._p_activate()
print ((Preloaded '%s' %s index btree (%d items in %.2fs))) % (
index_name, index_type, i, time.time() - start,
)
If you ever get an empty btree, you'll get an UnboundLocalError: 'i' here.
Drop the enumerate() trick and just use len(btree), it's efficient.
def preload_catalog(catalog):
Given a catalog, touch every persistent object we can find to
force
them to go into the cache.
start = time.time()
num_indices = len(catalog.items())
for i, (index_name, index) in enumerate(catalog.items()):
print ((Preloading index %2d/%2d '%s'...)) % (i+1,
num_indices, index_name,)
preload_index_btree(index_name, 'fwd', index._fwd_index)
preload_index_btree(index_name, 'rev', index._rev_index)
print ((Preloaded catalog! Took %.2fs)) % (time.time() - start)
And I run it on server start as follows (modified for the relevant parts; I
tried to make the example simple but it ended up needing a lot of parts).
This runs in a thread:
from util import zodb as Z
from util import zodb_query as ZQ
for i in xrange(3):
connwrap = Z.ConnWrapper('index')
print ((Preload #%d...)) % (i+1)
with connwrap as index_root:
ZQ.preload_catalog(index_root.index.catalog)
connwrap.close()
Every thread has its own in-memory ZODB object cache, but if you have
configured a persistent ZEO client cache, it should help.
Marius Gedminas
--
Never trust a computer you can't repair yourself.
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] API question
On Mon, Jan 14, 2013 at 01:32:07PM -0500, Tres Seaver wrote:
While working on preparation for a Py3k port, I've stumbled across a
fundamental issue with how ZODB structures its API. Do we intend that
client code do the following::
from ZDOB import DB, FileStorage
db = DB(FileStorage('/path/to/Data.fs'))
ZODB.FileStorage is a module, you can't call it.
ZODB.DB, much to my surprise, refers to the ZODB.DB.DB class. A
backwards compatibility thing maybe?
or use the module as a facade ::
import ZODB
db = ZODB.DB(ZODB.FileStorage.FileStorage('/path/to/Data.fs'))
This rings warning bells in my mind: if you're using the
ZODB.FileStorage module, you should import it directly:
import ZODB
import ZODB.FileStorage
db = ZODB.DB(ZODB.FileStorage.FileStorage('/path/to/Data.fs'))
I would actually prefer that clients explicitly import the intermediate
modules::
from ZDOB import DB, FileStorage
db = DB.DB(FileStorage.FileStorage('/path/to/Data.fs'))
(I'm not a fan of this style, but never mind that.)
or even better::
from ZDOB.DB import DB
# This one can even be ambiguous now
from ZODB.FileStorage import FileStorage
db = DB(FileStorage('/path/to/Data.fs'))
This is what I usually do.
I don't get the ambiguous comment. ZODB.DB is (currently) always the
class[1]. ZODB.FileStorage is always the module.
[1] I think (currently) the only way to refer to the ZODB.DB module is
to use sys.modules['ZODB.DB']:
import ZODB
ZODB.DB
class 'ZODB.DB.DB'
from ZODB import DB
DB
class 'ZODB.DB.DB'
import ZODB.DB
ZODB.DB
class 'ZODB.DB.DB'
The driver for the question is getting the tests to pass under both
'nosetests' and 'setup.py test', where the order of module imports etc.
can make the ambiguous cases problematic. It would be a good time to do
whatever BBB stuff we need to (I would guess figuring out how to emit
deprecation warnings for whichever variants) before releasing 4.0.0.
Can you demonstrate the ambiguity? As I've shown before, I was unable
to find it, at least with Python 2.x.
Marius Gedminas
--
We don't really understand it, so we'll give it to the programmers.
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 4.0.0a1 released
On Wed, Nov 07, 2012 at 04:45:59PM -0500, Jim Fulton wrote: Not to be confused with ZODB3! :) This is the first ZODB 4 release. Not to be confused with the old ZODB4 either, I suppose ;) http://wiki.zope.org/ZODB/ZODB4 Marius Gedminas -- A real friend isn't someone you use once and then throw away. A real friend is someone you can use over and over again. signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Storm/ZEO deadlocks (was Re: [Zope-dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB)
On Thu, Aug 30, 2012 at 11:19:22AM -0600, Shane Hathaway wrote: On 08/30/2012 10:14 AM, Marius Gedminas wrote: Here's the code to reproduce it: http://pastie.org/4617132 Updated version with more explicit logging and fewer unnecessary things: http://pastie.org/4630898 And here's the output: http://pastie.org/4631136 The deadlock happens in tpc_begin() in both threads, which is the first phase, AFAIU. AFAICS Thread #2 first performs tpc_begin() for ClientStorage and takes the ZEO commit lock. Then it enters tpc_begin() for Storm's StoreDataManager and blocks waiting for a response from PostgreSQL -- which is delayed because the PostgreSQL server is waiting to see if the other thread, Thread #1, will commit or abort _its_ transaction, which is conflicting with the one from Thread #2. Meanwhile Thread #1 is blocked in ZODB's tpc_begin(), trying to acquire the ZEO commit lock held by Thread #2. It looks like I mixed up the thread numbers when I was writing this up last night, i.e. in the above Thread #2 is the one running work1(), and Thread #1 is the one running work2(). Sorry about that. In my defense, that was the order in which they were printed out, which was the iteration order of the sys._current_frames() dict. So thread 1 acquires in this order: 1. PostgreSQL 2. ZEO Thread 2 acquires in this order: 1. ZEO 2. PostgreSQL SQL databases handle deadlocks by detecting and automatically rolling back transactions, while the transaction package expects all data managers to completely avoid deadlocks using the sortKey method. I haven't looked at the code, but I imagine Storm's StoreDataManager implements IDataManager. I wonder if StoreDataManager provides a consistent sortKey. The sortKey method must return a string (not an integer or other object) that is consistent yet different from all other participating data managers. Thread 1 (i.e. work2) acquires the PostgreSQL lock by issuing that DELETE statement, at which point we haven't started the transaction commit yet, on either thread. On the other hand, the second thread (work1) tries to lock PostgreSQL in the UPDATE statement that happens during store.flush() that happens during tpc_begin() that happens after it's already holding the ZEO lock. So maybe it would be enough to make Storm's StoreDataManager sort before ZEO always, so we always take PostgrsSQL locks before ZEO locks. For the record, adding a store.flush() before transaction.commit() inside work1() makes this particular instance of the deadlock go away (and one of the transactions fail with a TransactionRollbackError: could not serialize access due to concurrent update). Marius Gedminas -- The worst thing about going out is that you're not in your house. -- Kimiko Ross signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [announce] NEO 1.0 - scalable and redundant storage for ZODB
On Tue, Aug 28, 2012 at 06:31:05PM +0200, Vincent Pelletier wrote: On Tue, 28 Aug 2012 16:31:20 +0200, Martijn Pieters m...@zopatista.com wrote : Anything else different? Did you make any performance comparisons between RelStorage and NEO? I believe the main difference compared to all other ZODB Storage implementation is the finer-grained locking scheme: in all storage implementations I know, there is a database-level lock during the entire second phase of 2PC, whereas in NEO transactions are serialised only when they alter a common set of objects. This could be a compelling point. I've seen deadlocks in an app that tried to use both ZEO and PostgreSQL via the Storm ORM. (The thread holding the ZEO commit lock was blocked waiting for the PostgreSQL commit to finish, while the PostgreSQL server was waiting for some other transaction to either commit or abort -- and that other transaction couldn't proceed because it was waiting for the ZEO lock.) Marius Gedminas -- People who think, Oh this is a one-off, need to be offed, or perhaps politely removed from the project. -- George Neville-Neil signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Broken objects problem.
On Tue, Aug 07, 2012 at 08:46:58PM -0300, Matias Pansa wrote: I'm developing a web app using flask - zodb (zeo) and gunicorn as server , i'm facing the following problem : from a server that i wrote i'm recieving data from internet of some gps and the same server store the data via zeo client , here everything is ok , i can check the data with vista.py script tha simply connects via zeo client and get the results from the database , the data class is Posicion and represents a gps position in time , all the objects are stored correctly.i check almos When i try to do the same thing from inside tha flask app running gunicorn all i get in the resultset of data from zeo are broken objects with no class so i can't access any attributes or methods of the class Posicion . My question is , could this problem be related to zeo or zodb ? , or i keep looking to gunicorn/flask problem ? , i check almost everything that's why i'm asking here. A broken object means the class could not be imported using its full dotted name (package.module.ClassName). Check your Python path. I may also be worth checking if the objects are saved using the correct package/module, and not, say, '__main__'. Marius Gedminas -- Never be afraid to tell the world who you are. -- Anonymous signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Is any function called when an object is loaded from the database?
On Tue, Jun 19, 2012 at 01:54:21PM -0400, Claudiu Saftoiu wrote: Ideally I could do something like this: class Foo(Persistent): def __init__(self, a): self.a = a self.b_cache = PersistentDict() def __just_loaded__(self): if not hasattr(self, 'b_cache'): self.b_cache = PersistentDict() def calc_it(self, b): if b in self.b_cache: return self.b_cache[b] res = expensive_function(a, b) self.b_cache[b] = res return res That is, a function called whenever the object is loaded, that does all the necessary backwards-compatibility work right there. It separates the backwards-compat code cleanly, and also only updates the objects as-needed... though still a minor performance hit as it does the check each time the object is loaded. Is there a way to do that last option? Yes, you can override __setstate__: def __setstate__(self, state): super(Foo).__setstate__(self, state) if not hasattr(self, 'b_cache'): self.b_cache = PersistentDict() This assumes you haven't been defining other pickling protocol functions like __reduce__, and so cPickle -- which is the foundation of ZODB -- uses the usual __getstate__/__setstate__ pair. What's the best practice for this sort of thing, in general? It depends. :-) If the new attribute is immutable, you can set it as a class attribute. If it's easy to locate all Foo objects in your DB, you may want to write a generation script (see http://pypi.python.org/pypi/zope.generations) and do the migration once, instead of paying the cost of checking for missing attributes on every single object load. It's also best to avoid write-on-read semantics, because those tend to cause database growth and increase the chances of getting ConflictErrors. Marius Gedminas -- An expert is a person who has made all the mistakes that can be made in a very narrow field. -- Niels Bohr signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Is any function called when an object is loaded from the database?
On Tue, Jun 19, 2012 at 10:15:03PM +0200, Vincent Pelletier wrote: Le mardi 19 juin 2012 19:54:21, Claudiu Saftoiu a écrit : def calc_it(self, b): if not hasattr(self, 'b_cache'): self.b_cache = PersistentDict() If you were to use this, you would probably prefer to do try: self.b_cache except AttributeError: self.b_cache = PersistentDict() (assigning the result to a vairable, etc) as this is faster when b_cache is actually here (which is much more likely than not). Also, it avoids hiding too many exceptions, as with hasattr. In the times of ReadConflictError it was quite important (at least to stop processing earlier on a doomed transaction). I'm not sure how important this is nowadays - I just kept the habit of avoiding hasattr on persistent objects. The former is OK, but if I have multiple functions that want to use the new functionality I'll have to have the code all over, and it won't be obviously separated. Maybe a @property taking care of creating the actual PersistentDict if not present, then ? zope.cachedescriptors.property.Lazy would work even better: @Lazy def b_cache(self): return PersistentDict() Marius Gedminas -- If you have to run heating in winter, you don't own enough computers. signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to cause commit with non-PersistentDict?
On Tue, Jun 19, 2012 at 04:22:43PM -0400, Claudiu Saftoiu wrote:
Hello all,
Say I have a:
class Foo(Persistent):
def __init__(self, bar):
self.my_dict = PersistentDict({'keyis': bar})
def update_it(self, bar):
self.my_dict['keyis'] = bar
If I want to use a `dict`, instead (it seems it might be faster for my
larger example),
I doubt that.
(OTOH if you have a large number of keys, consider using OOBTree instead
of PersistentDict.)
how would I cause a change to the dict to
be committed? Is there any way other than this?
class Foo(Persistent):
def __init__(self, bar):
self.my_dict = {'keyis': bar}
def update_it(self, bar):
self.my_dict['keyis'] = bar
self.my_dict = dict(self.my_dict)
Yes: either
def update_it(self, bar):
self.my_dict['keyis'] = bar
self.my_dict = self.my_dict
or
def update_it(self, bar):
self.my_dict['keyis'] = bar
self._p_changed = True
I recommend against this pattern in general: it's too easy to forget the
_p_changed bit and end up with bugs that are hard to notice. Because
ZODB caches object instances, nonpersistent subobject modifications live
on in memory until those objects get flushed (or Zope gets restarted),
so you only notice you've a bug in your code when changes start
mysteriously disappearing a few days after they're made.
Marius Gedminas
--
The Feynman Problem Solving Algorithm:
1) Write down the problem.
2) Think very hard.
3) Write down the solution.
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] unable to import ZODB: class ConflictError, AttributeError
from ZODB.Connection import Connection File /usr/local/lib/python2.6/dist-packages/ZODB3-3.10.5-py2.6-linux-x86_64.egg/ZODB/Connection.py, line 33, in module from ZODB.blob import Blob, rename_or_copy_blob, remove_committed_dir File /usr/local/lib/python2.6/dist-packages/ZODB3-3.10.5-py2.6-linux-x86_64.egg/ZODB/blob.py, line 35, in module from ZODB.POSException import POSKeyError File /usr/local/lib/python2.6/dist-packages/ZODB3-3.10.5-py2.6-linux-x86_64.egg/ZODB/POSException.py, line 71, in module class ConflictError(POSError, transaction.interfaces.TransientError): AttributeError: 'module' object has no attribute 'TransientError' Can you do import transaction transaction.__file__ ? Marius Gedminas -- The IQ of the group is the lowest IQ of a member of the group divided by the number of people in the group. signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] how to get root obj from a new transaction?
On Wed, May 02, 2012 at 12:49:32PM -0400, Claudiu Saftoiu wrote:
I'm using a thread to do some server-side work. The thread will be
asked by different requests to do the same thing at the same time, so
I want the thread to do all the work once and return the data to the
requests. The problem is that the requests each have their own
transaction and the thread essentially has none.
Right.
I can communicate between the two only using identifiers - not
persistent objects - that way the thread can process data in a
different transaction than the requests yet still return a meaningful
reply.
Also correct: you should not share instances of persistent objects between
threads.
However, the thread has to start a new transaction each time it
processes something - which I know how to do:
while True:
#wait until asked to do something
import transaction
transaction.begin()
However, the thread needs access to the root object in order to turn
the identifiers gotten from the requests into persistent objects...
how would I go about accessing the root object in such a circumstance?
You need to pass the database object to the thread, and the thread needs
to open a connection (connection = db.open()). Then connection.root()
will give you the root object (or you could pass OIDs to the thread and
use connection.get(oid) to find the objects you need to work with).
Don't forget to commit or abort the transaction, and also don't forget
that you may need to implement some kind of retry logic if commit()
raises a ConflictError due to conflicting updates.
Marius Gedminas
--
main(k){float i,j,r,x,y=-16;while(puts(),y++15)for(x
=0;x++84;putchar( .:-;!/)|IH%*#[k15]))for(i=k=r=0;
j=r*r-i*i-2+x/25,i=2*r*i+y/10,j*j+i*i11k++111;r=j);}
/* Mandelbrot in ASCII. */
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used
On Fri, Apr 20, 2012 at 10:06:40AM -0400, Claudiu Saftoiu wrote: Ah, so it seems that, as I leave the server running longer longer, more more threads are taken up with a `.recv()` call. I think one of my clients opens requests and does not read them/close them. Does not write, rather. Eventually all the threads are blocking in that fashion. I will fix my clients. But, is there a server-side fix to this (again, using Repoze.BFG)? Additional info: in particular the blocked threads' stack dumps look like this: Thread 140605868680960: File /usr/lib/python2.6/threading.py, line 504, in __bootstrap self.__bootstrap_inner() File /usr/lib/python2.6/threading.py, line 532, in __bootstrap_inner self.run() File /usr/lib/python2.6/threading.py, line 484, in run self.__target(*self.__args, **self.__kwargs) File /home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py, line 878, in worker_thread_callback runnable() File /home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py, line 1052, in lambda lambda: self.process_request_in_thread(request, client_address)) File /home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py, line 1068, in process_request_in_thread self.finish_request(request, client_address) File /usr/lib/python2.6/SocketServer.py, line 322, in finish_request self.RequestHandlerClass(request, client_address, self) File /usr/lib/python2.6/SocketServer.py, line 617, in __init__ self.handle() File /home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py, line 442, in handle BaseHTTPRequestHandler.handle(self) File /usr/lib/python2.6/BaseHTTPServer.py, line 329, in handle self.handle_one_request() File /home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py, line 437, in handle_one_request self.wsgi_execute() File /home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py, line 287, in wsgi_execute self.wsgi_start_response) File /home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py, line 21, in __call__ result = self.next_app(environ, start_response) File /home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/cachecleanup.py, line 25, in __call__ return self.next_app(environ, start_response) File /home/tsa/env/lib/python2.6/site-packages/repoze/retry/__init__.py, line 65, in __call__ chunk = original_wsgi_input.read(rest) File /home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py, line 474, in read data = self.file.read(length) File /usr/lib/python2.6/socket.py, line 377, in read data = self._sock.recv(left) Was my assessment of the situation accurate? It appears so. Here's the code of repoze.retry: https://github.com/repoze/repoze.retry/blob/master/repoze/retry/__init__.py Line numbers differ somewhat; the original_wsgi_input.read(rest) is on line 71 on that github page right now. It does look like this is the bit that reads the HTTP request from the client. Something to time out the connection after 60 seconds or so if nothing has happened? You're using paste.httpserver. It has support for killing hung threads: http://pythonpaste.org/modules/httpserver.html#paste.httpserver.server_runner has a list of options. You could also put nginx in front, to take care of ill-behaved HTTP clients. Marius Gedminas -- Never be afraid to tell the world who you are. -- Anonymous signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction Attempts class
On Wed, Mar 28, 2012 at 05:06:37PM -0400, Jim Fulton wrote: On Wed, Mar 28, 2012 at 4:37 PM, Chris McDonough chr...@plope.com wrote: On Wed, 2012-03-28 at 14:21 -0400, Jim Fulton wrote: ... A decorator for running some code in the context of a txn and retrying retryable exceptions would be nice higher level behavior. Â I'd be willing to do this work over this weekend. Cool. Don't forget the transaction note part. :) Too many transactions in our apps don't have notes, especially transactions that happen outside of web requests. I've started using transaction.get().note() liberally only after I was able to see the notes in ZODBBrowser (and became annoyed by transactions that didn't have notes). That reminds me, I need to patch ZODBBrowser to show extended info (such as the request type and URL). Marius Gedminas -- Shift happens. -- Doppler signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Error installing ZODB (transaction 1.2.0)
On Thu, Mar 29, 2012 at 07:56:54AM +0200, Thierry Florac wrote: Le Wed, 28 Mar 2012 19:21:51 +, Bahr, Eric eric.b...@northwestern.com a �crit: I am using the following that seem to be working fine, and I would like to continue using them, but don't know how to transfer them to another computer: ZODB-3.10.0b1-py2.5-win32.egg zope.event-3.5.1-py2.5.egg zope.interface-3.8.0-py2.5-win32.egg transaction-1.1.1-py2.5.egg zc.lockfile-1.0.0-py2.5.egg zconfig-2.9.0-py2.5.egg zdaimon-2.0.4-py2.5.egg You should probably use zc.buildout. It will allow you to easily define the packages (and their versions) you need... Or you could use a pip requirements.txt file: $ cat requirements.txt zope.event==3.5.1 transaction==1.1.1 zc.lockfile==1.0.0 ... and install them all with 'pip install -r requirements.txt'. (In a virtualenv, obviously, because cluttering site-packages does not scale.) Buildout is much more capable, but has a somewhat steeper learning curve. Marius Gedminas -- If you are angry with someone, you should walk a mile in their shoes... then you'll be a mile away from them, and you'll have their shoes. signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] database conflict fail
On Thu, Mar 22, 2012 at 04:13:34PM -0400, Claudiu Saftoiu wrote:
On Thu, Mar 22, 2012 at 2:45 PM, Vincent Pelletier
plr.vinc...@gmail.comwrote:
Le jeudi 22 mars 2012 18:23:47, Claudiu Saftoiu a écrit :
Ahh, now that looks promising. Are there any particularly good places to
get documentation on that sort of thing? All I see when I google are
mailing list archives.
See ZODB/ConflictResolution.txt . Disclaimer: I didn't read it. I read the
code - but I expect the text file to be easier to assimilate :) .
BTrees/Length.py:Length class implements a simple _p_resolveConflict (yours
will be even simpler).
Thanks, I will take a look.
There should be some details in the exceptoin itself. Like, oids, currently
commited TID and the TID transaction started with. root()._p_jar[the_oid]
will
get you the object. Then the hard part starts: guess where in the object
tree
that object is. If you have varied classes, and significant data on your
persistent instances, it will be easy.
Ok, I just got a ConflictError:
ConflictError: database conflict error (oid 0x139c35, class
BTrees.OOBTree.OOBucket, serial this txn started with 0x03954ed053c0ff88
2012-03-22 16:48:19.629820, serial currently committed 0x03954f996d61c944
2012-03-22 20:09:25.636401)
in my paster shell I do:
In [14]: root._p_jar[0x139c35]
OIDs are 64-bit numbers internally encoded as 8-byte-long binary strings.
Use
!from ZODB.utils import p64
p root._p_jar.get(p64(0x139c35))
or http://pypi.python.org/pypi/zodbbrowser
However, this causes:
In [14]: root._p_jar[0x139c35]
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (85, 0))
Where did the EOF come from?
(This is why I don't trust ipython.)
---
TypeError Traceback (most recent call last)
/home/tsa/sports/ipython console in module()
/home/tsa/env/lib/python2.6/site-packages/ZODB/Connection.pyc in get(self,
oid)
246 return obj
247
-- 248 p, serial = self._storage.load(oid, '')
249 obj = self._reader.getGhost(p)
250
/home/tsa/env/lib/python2.6/site-packages/ZEO/ClientStorage.pyc in
load(self, oid, version)
813 self._lock.acquire()# for atomic processing of
invalidations
814 try:
-- 815 t = self._cache.load(oid)
816 if t:
817 return t
/home/tsa/env/lib/python2.6/site-packages/ZEO/cache.pyc in call(*args, **kw)
141 inst._lock.acquire()
142 try:
-- 143 return self.func(inst, *args, **kw)
144 finally:
145 inst._lock.release()
/home/tsa/env/lib/python2.6/site-packages/ZEO/cache.pyc in load(self, oid)
487 @locked
488 def load(self, oid):
-- 489 ofs = self.current.get(oid)
490 if ofs is None:
491 self._trace(0x20, oid)
/home/tsa/env/lib/python2.6/site-packages/ZODB/fsIndex.pyc in get(self,
key, default)
123
124 def get(self, key, default=None):
-- 125 tree = self._data.get(key[:6], default)
126 if tree is default:
127 return default
TypeError: 'int' object is unsubscriptable
This makes more sense than the EOF error above.
Marius Gedminas
--
I am monolingual and English is the ling I mono.
-- James Nicoll
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] database conflict fail
On Thu, Mar 22, 2012 at 11:07:09PM +0200, Marius Gedminas wrote: On Thu, Mar 22, 2012 at 04:13:34PM -0400, Claudiu Saftoiu wrote: Ok, I just got a ConflictError: ConflictError: database conflict error (oid 0x139c35, class BTrees.OOBTree.OOBucket, serial this txn started with 0x03954ed053c0ff88 2012-03-22 16:48:19.629820, serial currently committed 0x03954f996d61c944 2012-03-22 20:09:25.636401) in my paster shell I do: In [14]: root._p_jar[0x139c35] OIDs are 64-bit numbers internally encoded as 8-byte-long binary strings. Use !from ZODB.utils import p64 p root._p_jar.get(p64(0x139c35)) Sorry, I shouldn't use pdb notation on the interactive Python prompt. from ZODB.utils import p64 root._p_jar.get(p64(0x139c35)) or http://pypi.python.org/pypi/zodbbrowser Marius Gedminas -- IBM motto: TEN vowels? Don't you know vowels are scrd? -- Linus Torvalds signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB via Pipe/Socket
On Tue, Mar 20, 2012 at 12:52:07PM -0400, Adam Tauno Williams wrote: It is possible to open a ZODB in a thread and share it to other threads via a filesystem socket or pipe [rather than a TCP conntection]? You don't need either if you use threads. Create a ZODB.DB.DB(), have each thread invoke db.open() to get a connection. If you have multiple *processes* rather than threads, use ZEO. ZEO works fine over Unix domain sockets (this is, in fact, my preferred method of deployment). Marius Gedminas -- There is a 70% probability of tomorrow. (Actual weatherman quote, 1988) signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] How to update an object in a multithreading application?
On Mon, Mar 19, 2012 at 02:04:34PM -0300, Sebastian Wain wrote: I am updating an item in a separated thread but when it finishes the new value is not updated in the same object in the main thread. In the example below root['counter'] starts in 0, is incremented in the new thread but it remains in 0 in the main thread. #!/usr/bin/python from ZODB.FileStorage import FileStorage from ZODB.DB import DB from BTrees.OOBTree import OOBTree import transaction from threading import Thread import persistent class MyThread(Thread): def __init__(self, db): Thread.__init__(self) self.db = db def run(self): self.connection = self.db.open(transaction.TransactionManager()) This looks unnecessarily complicated. Just use db.open(), transaction.begin(), transaction.commit() -- these functions always use a thread-local transaction. self.root = self.connection.root() print Starting thread self.connection.transaction_manager.begin() self.root[counter] += 1 self.connection.transaction_manager.commit() print self.root['counter'] =, self.root['counter'] filename = 'test.fs' storage = FileStorage(filename) db = DB(storage) connection = db.open() root = connection.root() if 'counter' not in root: transaction.begin() root['counter'] = 0 transaction.commit() thread = MyThread(db) thread.start() thread.join() print *** After *** As Alan said, you need a connection.sync() here. I believe calling transaction.begin() again at this point would also do that (just remember that transaction.begin() implicitly aborts the previous transaction, so don't do that if you have uncommitted changes you want to keep). print root['counter'] =, root['counter'] Marius Gedminas -- This is, or so I'm told, a classic book. I can see why. It is long and it requires a great effort to finish it. If you go through a great effort to suffer for a long time, you are allowed to sneer at people who can't be bothered. That's the surest sign of classic art. -- Lars Wirzenius signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack error
On Thu, Feb 09, 2012 at 02:26:10AM +0100, Kaweh Kazemi wrote: Thanks a lot for the investigation, Marius. That was awesome. Seems to me, I need to upgrade to Python 2.7 and be done with the problem then. :/ I wonder, though, if ZODB ought to use the 'noload' method of cPickle.Unpickler, which is undocumented and broken on 2.6 (and gone on 3.x, AFAIU). Incidentally, since you use cross-database references, please make sure they continue to work after you pack your storage. I've lost data that way (the ZODB garbage collector doesn't see references that exist in other storages, and can assume objects are garbage when it shouldn't). Packing with GC disabled ought to be safe. Cross-database references are only one-way in our implementation. users storage is referencing objects in the game storage - never vice versa / no circles. Also we make sure we never delete any objects in *any* storage, just disable them on an application level. This worked fine so far. Are you seeing any problems with this? I don't use cross-database references. My only encounter with them was on Zope 2.x: I uploaded a new version of a website from a .zexp file into /temp_folder, tested it briefly there, then used the ZMI to move it into /. It worked fine for a day, and was broken the next day. Turns out /temp_folder in Zope 2 uses a different backend storage, and moving the folder to root just created a cross-db reference, but didn't move the objects across. Then the garbage collector for the /temp_folder storage noticed that my site was not directly reachable from /temp_folder's root dict, and collected it away as garbage. I do now know if that bug is still present in any version of Zope or ZODB; I just learned not to trust cross-db references. Once burned twice shy, as they say. I'm not saying you will encounter any garbage-collection related problems; I'm saying I would double-check to be sure if I was in your position. Disabling GC during pack works (and was a workaround for a while), but the difference between non-GC (4G) and GC pack (1GB) are 3GB, which is quite a lot and growing too fast. We just don't need all the references from old object states to persist. Packing always removes old object states. Garbage collection also removes new object states for objects that are no longer reachable. Does zeopack consider cross-database references correctly when it checks for reachability? I would hope so, but I don't really know. It would only work if you're packing all the storages together, presumably. Marius Gedminas -- Well, there's a quantum computer that can factor 15, so those of you using 4-bit RSA should worry. -- Bruce Schneier signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack error
that way (the ZODB garbage collector doesn't see references that exist in other storages, and can assume objects are garbage when it shouldn't). Packing with GC disabled ought to be safe. Marius Gedminas -- The world is really obsessing over the UI preferences of the person who gave us git? -- Matthew Garrett signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeopack error
On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
Recap: last week I examined problems I had packing our 4GB users
storage.
...
unp = pickle.Unpickler(f)
unp.persistent_load = lambda oid: 'persistent reference %r' % oid
pprint.pprint(unp.load())
{'data': {persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\tT', class '__main__.Tool')]: 1,
persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', class
'__main__.EnergyPack')]: 1}}
Those look like cross-database references to me.
The original error (aaaugh Mutt makes it hard for me to look upthread
while I'm writing a response) was something about non-hashable lists?
Looks like a piece of code is trying to put persistent references into a
dict, which can't possibly work in all cases.
...
During my checks I realized that running the pack in a Python 2.7
environment (using the same ZODB version - 3.10.3) works fine, the
pack reduces our 4GB storage to 1GB. But our production server uses
Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
had been done on OS X 10.7.3 - 64bit, and the production server is
Debian Squeeze 32bit).
I've no idea why running the same ZODB version on Python 2.7 instead of
2.6 would make this error go away.
Duh! The code that fails is in the standard library -- in the cPickle
module:
Traceback (most recent call last):
...
File
/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py,
line 328, in findrefs
return self.referencesf(self._file.read(dh.plen))
File
/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py,
line 630, in referencesf
u.noload()
TypeError: unhashable type: 'list'
Since the bug is in the stdlib, it's not surprising that the newer
stdlib cPickle from Python 2.7 fixes it.
Looks like it was this one: http://bugs.python.org/issue1101399
(hg changeset 56532:d0f005e6fadd)
When I diff Modules/cPickle.c between the 2.6 and 2.7 branches in the hg
repository, I see that noload() handles the APPEND, APPENDS, SETITEM and
SETITEMS differently in 2.7. hg blame attributes that change to 56532.
The SETITEM/SETITEMS opcode in the pickle now no longer tries to
actually stuff anything into any dicts, which is why the list never ends
up being hashed, which explains the error and why it is gone.
The fix is present in 2.7, but not in 2.6
That was fun.
Marius Gedminas
--
Critical sections are Windows's lightweight alternative to mutexes, thus
cleverly getting more use out of a previously well-defined phrase by using it
to mean something related to but slightly different from the earlier general
understanding
-- Verity Stob
signature.asc
Description: Digital signature
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] where does 32it vs 64bit matter?
On Tue, Jan 03, 2012 at 06:54:32AM +, Chris Withers wrote: On 21/12/2011 20:12, Alan Runyan wrote: Are there any migration steps or minimum ZODB/RelStorage versions needed to make this step? We run 32bit windows clients against 64bit linux zeo server; works fine. Good to know, but my concern is taking a system which is currently running 32-bit clients against RelStorage running on a MySQL on a 64-bit server and replacing the 32-bit clients with 64-bit clients. Is that going to cause any issues? Has anyone done any similar migrations? With RelStorage - no. With FileStorage - yes. No issues after moving a few Data.fs'es from 32-bit to 64-bit Linux machines. Marius Gedminas -- Special bonus feature: absolutely nowhere in RetchMail's code is there an arbitrary 3-second sleep(). Wow! What other mail retriever can say that? (Hint: not fetchmail.) -- http://alumnit.ca/wiki/index.php?page=RetchMail signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] A couple of repozo buglets
On Wed, Dec 07, 2011 at 03:00:10AM +0200, Marius Gedminas wrote: With complimentary automated functional test cases and a README at https://github.com/mgedmin/repozo-testcases Bug #1: insufficient error detection. You can truncate a *.deltafs in the middle, and repozo -R doesn't notice -- it goes on to produce a corrupt Data.fs. Filed as https://bugs.launchpad.net/zodb/+bug/906057 Bug #2: two frequent runs of repozo -B may cause repozo -R to silently produce a corrupt Data.fs. Filed as https://bugs.launchpad.net/zodb/+bug/906058 I'm not opposed to spending some time working on this, if somebody can provide some hand-holding on IRC (I've motivation issues due to burnout). The offer still stands. Marius Gedminas -- This company has performed an illegal operation and will be shut down. If the problem persists, contact your vendor or appeal to a higher court. signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] A couple of repozo buglets
With complimentary automated functional test cases and a README at https://github.com/mgedmin/repozo-testcases Bug #1: insufficient error detection. You can truncate a *.deltafs in the middle, and repozo -R doesn't notice -- it goes on to produce a corrupt Data.fs. AFAIU the *.dat file contains enough information to do basic sanity checking for the sizes of the various *.deltafs files. It'd be nice if repozo -R noticed and aborted with a loud complaint when one of the files is truncated. Checksum checking would be even nicer, but, perhaps, expensive, and so maybe ought to require an extra command line option to enable. To reproduce: git clone git://github.com/mgedmin/repozo-testcases.git cd repozo-testcases make test2 Bug #2: two frequent runs of repozo -B may cause repozo -R to silently produce a corrupt Data.fs. Discovered while I was writing the above test case. If there's a timestamp collision between the original .fs and the first .deltafs, repozo -R produces a truncated Data.fs. (If there's a timestamp collision between two .deltafs files, repozo -B aborts with an error, which is fine by me.) To reproduce: git clone git://github.com/mgedmin/repozo-testcases.git cd repozo-testcases make test1 I've found the first bug in ZODB3 3.9.7, and reproduced both with the latest ZODB3 3.10.5. I'm not opposed to spending some time working on this, if somebody can provide some hand-holding on IRC (I've motivation issues due to burnout). Hm, this email is probably more descriptive than the README.txt. Marius Gedminas -- The clothes have no emperor. -- C.A.R. Hoare, commenting on ADA. signature.asc Description: Digital signature ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] I would like to know if api exists nowadays to get list of all objects which where modified in a transaction ?
On Fri, Jul 29, 2011 at 03:21:23PM +0200, Pedro Ferreira wrote: some time ago, someone ask how to get list of all objects which where modified in a transaction : http://article.gmane.org/gmane.comp.web.zope.zodb/5734 I do not know if this is what you are looking for: https://gist.github.com/1113786 It's a small thing that i hacked some months ago, and that provides a detailed log on what is going on behind the scenes (client-side). I've a smaller hack in zodbbrowser (sorry for the long URL, launchpad sucks): http://bazaar.launchpad.net/~zodbbrowser-dev/zodbbrowser/trunk/view/head:/src/zodbbrowser/browser.py#L84 Basically, if you don't mind peeking in private attributes of transaction and connection objects, you can easily find out what objects were added/modified. I wouldn't hesitate use this for debugging, but I would not build business logic on top of this. Marius Gedminas -- It also makes sense in the case of 1872, where most of Horace Greenly's electors came to the sensible conclusion that the death of their candidate made him unfit for office. -- matthewwdaly explains the workings of the US Electoral College signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] TypeError: ('object.__new__(SyncedLanguages) is not safe, use Persistence.Persistent.__new__()', ...
On Fri, Dec 31, 2010 at 08:39:22AM +0100, Godefroid Chapelle wrote: Le 31/12/10 04:08, Marius Gedminas a écrit : On Fri, Dec 31, 2010 at 02:02:36AM +0100, Godefroid Chapelle wrote: I have been fighting all day to understand a bug with the unregistration of a utility from a local site. After unregistration, some instance is left in the _subscribers list of the utilities AdapterRegistry in my local site. Sounds like https://bugs.launchpad.net/zope.component/+bug/551049 ... Actually, half of it is based on identity, and half on equality; when these disagree, you end up with inconsistent state. Yup Your example is a good argument for making both halves use equality rather than identity. ... I think a more fruitful approach would be to fix the zope.component bug and then unregister the utility in the normal way. Marius, did you already try to fix zope.component ? No. I didn't have an opinion how to fix it (make both pieces of code use identity or equality), and I was a bit afraid to touch the internals in case I broke something. And it was easy to work around in my code, so I just chickened out :( Is it because it is hard that you accepted the Won't fix ? Or because of lack of time ? The wontfix is for the obsolete monolithic zope3 package. The bug is still open against zope.component. Although it may need to be fixed in the internals of the adapter registry which lives, I think, in zope.interface? I forgot most of the details :( Alternatively, you could hack up a __new__ method to always return the same instance, so you can unregister it cleanly before removing the code. This will not work if you have more than one instance in the DB. Could that __new__ method insert Persistent as a base class if not present ? No. In order to change from non-persistent to persistent, you need to activate and dirty all objects that have references to your nonpersistent object; it's not enough to change just the object itself. Marius Gedminas -- The death rate on Earth is: (computing) One per person. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] TypeError: ('object.__new__(SyncedLanguages) is not safe, use Persistence.Persistent.__new__()', ...
On Fri, Dec 31, 2010 at 02:02:36AM +0100, Godefroid Chapelle wrote:
I have been fighting all day to understand a bug with the unregistration
of a utility from a local site.
After unregistration, some instance is left in the _subscribers list of
the utilities AdapterRegistry in my local site.
Sounds like https://bugs.launchpad.net/zope.component/+bug/551049
This avoids me to remove the code as it does leave a broken object.
I have come to the conclusion that the code that first registered the
utility is the culprit : the utility itself is not persistent.
It seems that the utility instance is unserialized at least two times
during unregistration, which leads (if I understand well) to two
different objects in memory.
This breaks the unsubscription algorithm that is based on object identity.
Actually, half of it is based on identity, and half on equality; when
these disagree, you end up with inconsistent state.
Your example is a good argument for making both halves use equality
rather than identity.
I naively tried to introduce the Persistent base class into the utility
calss. When I try to access the utility instance again, I get the error
mentioned in the subject of this mail :
TypeError: ('object.__new__(SyncedLanguages) is not safe, use
Persistence.Persistent.__new__()', function _reconstructor at
0x1004297d0, (class
'Products.LinguaPlone.vocabulary.SyncedLanguages', type 'object', None))
If I understand well, the a posteriori introduction of Persistent cannot
happen alone.
Can anyone tell me what I should do add to the class I try to make it
actually persistent ?
I think a more fruitful approach would be to fix the zope.component bug
and then unregister the utility in the normal way.
Alternatively, you could hack up a __new__ method to always return the
same instance, so you can unregister it cleanly before removing the
code. This will not work if you have more than one instance in the DB.
Fixing up _subscribers directly is also a possible workaround, if you're
feeling brave.
Marius Gedminas
--
/* Intel provided a special instruction to clear the TS bit for people too cool
* to use write_cr0() to do it. This clts instruction is faster, because all
* the vowels have been optimized out. */
-- lguest source code
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 32-bit vs 64-bit
On Thu, Nov 18, 2010 at 03:25:17PM +, Chris Withers wrote: Hi All, Are there any intricacies with migrating a ZODB-backed Zope 2 app from 32-bit to 64-bit machines? I haven't noticed any problems, when I moved my personal website from a 32-bit machine to a 64-bit one. About the only noticeable difference -- other than the obvious memory growth -- is that string hashes are computed differently, which changes the ordering of dict.keys(). Thankfully, pickles do not rely on that implementation detail. Marius Gedminas -- It's possible to describe anything in mathematical notation. I recall seeing some paper once in which someone had created a mathematical description of C. (I forget whether or not this included the preprocessor.) As an achievement, this is somewhat like building a full-size model of the Eiffel Tower out of tongue depressors. It's clearly not the act of a talentless man, but you have to wonder what he said when he applied for his grant. -- Mencius Moldbug signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: getting keep-history wrong results in silent death
On Thu, Nov 04, 2010 at 08:13:14AM -0600, Shane Hathaway wrote: On 11/04/2010 06:28 AM, Chris Withers wrote: I was having trouble starting a relstorage 1.4.0-backed Zope instance this morning. The symptoms were that it would get going fine, until the point it tried to open the storages, at which point it would simply restart with no logging as to why. Turns out I had keep-history set to true whereas the database have been created with it set to false. Should this not result in an exception being logged rather than just a silent death that causes zdaemon to lamely attempt to restart the instance? It outputs an error using the logging module and raises an exception. Are you saying some code somewhere is nastily swallowing that exception and suppressing the log? In the past I've been burned a few times by zdaemon discarding all stdout and stderr output from the child process, which resulted in this kind of child keeps restarting, sysadmin has no clue why if an exception happens during startup. Make sure your zdaemon.conf has runner ... transcript $LOGDIR/transcript.log ... /runner or something like that. Marius Gedminas -- If it wasn't for C, we'd be using BASI, PASAL and OBOL signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Default comparison considered harmful in BTrees.
On Mon, Oct 25, 2010 at 11:14:21PM +0100, David Glick wrote: On 10/25/10 11:07 PM, Jim Fulton wrote: On Mon, Oct 25, 2010 at 5:58 PM, David Glick davidgl...@groundwire.org wrote: On 10/25/10 10:51 PM, Jim Fulton wrote: I'm inclined to treat the use of the comparison operator inherited from object in BTrees to be a bug. I plan to fix this on the trunk. I'm tempted to fix this in 10.1. This change would make it impossible to add keys to BTrees or buckets or to add items to BTree-based sets if the key or items inherits it's comparison from object. This would only apply to instances of new-style classes, including persistent objects. (It wouldn't affect old-style-class instances, which are too hard to introspect.) Thoughts? I like this. The motivation here is that the comparison inherited from object compares the objects' memory locations, which is not stable beyond deactivation for persistent objects, and therefore not suitable for use as a BTree key. Correct? Yup. Thanks for clarifying. Preventing people from making that mistake sounds like a good thing to me. It will likely reveal that current applications are broken. This will likely cause some pain. Where previously, apps simply lost data, now they'll error. I'm afraid that this might be too disruptive for a bug-fix release. Maybe add it as an option defaulted to off? So that app developers who want to check whether they have this problem can easily do so, and have the extra protection once they've fixed any issues? Or perhaps make it emit DeprecationWarnings, but continue working. Then make it a fatal error in the next minor/major release. Marius Gedminas -- Include me out. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Issue trying to remove elements in OOTreeSet
On Fri, Sep 24, 2010 at 04:14:34PM +0200, César Muñoz wrote: I see... That explains the issue, after implementing the __cmp__ method everything is now working perfectly. Just note that if you have any valuable preexisting data, it may have been stored incorrectly, using the old default __cmp__ implementation. Dump it using items() and add it one-by-one to an empty OOTreeSet to recover. Marius Gedminas -- Anybody who doesn't cut his speed at the sight of a police car is probably parked. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Issue trying to remove elements in OOTreeSet
On Fri, Sep 24, 2010 at 11:09:58AM -0400, Jim Fulton wrote:
I wasn't proposing to change anything. :) Breaking inheritance to avoid
inheriting a bad cmp seems rather too tricky. (I assume there isn't
a default comparison in Python 3, although I haven't checked.)
I suppose that the BTree implementation could check for object's
bad comparison operator and fail if it sees it. (Details omitted. :)
This obviously wouldn't happen in 3.10, but might be a fun project for
someone in 3.11.
What would you think of adding
def __lt__(self, other):
if not isinstance(other, Persistent):
raise ValueError('cannot compare Persistent with %s' % repr(other))
if self._p_oid is None or other._p_oid is None:
raise ValueError('cannot compare objects that have no persistent
IDs yet; call connection.add()')
return self._p_oid other._p_oid
# define __le__, __gt__, __ge__ in a similar way
to persistent.Persistent?
Marius Gedminas
--
The old political syllogism something must be done: this is something:
therefore this will be done appears to be at work here, in spades.
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Issue trying to remove elements in OOTreeSet
On Thu, Sep 23, 2010 at 03:54:59PM +0200, César Muñoz wrote: That said, I'm having some problems when removing a chat room from the OOTreeSet. Sometimes (and this is weird, sometimes I may get the error, some others not) I get a KeyError exception when I try to remove the chat room. This is exactly the line that causes the problem: * root['indexByConf'][confId]['chatrooms'].remove(chatroom)* ... However, if you use the keys() method it will work, and if you use the operator in with a list of the OOTreeSet it will work also. Take a look at what I got from the interpreter: * chatroom* MaKaC.plugins.InstantMessaging.handlers.Chatroom object at 0x22ff646c It sounds as if your Chatroom class doesn't implement comparison operators (__lt__ and friends) correctly. If you want to store objects in an OOTreeSet, those objects must have a well-defined ordering that doesn't change when the objects are loaded back from the ZODB and end up at a different memory address. The default comparison provided by Python doesn't cut it. So: does your Chatroom define __cmp__ or __lt__, __le__, __gt__, __ge__, __eq__, __ne__ methods? Marius Gedminas -- Writing setattr hooks properly is a black art. Writing persistent setattr hooks is more like hearding bees blindfolded... -- Casey Duncan signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RFC: deprecate transaction user and description fields in favor of extened info and simplify extended info API
On Thu, Sep 23, 2010 at 10:53:03AM -0400, Stephan Richter wrote: On Thursday, September 23, 2010, Jim Fulton wrote: Comments? Sounds good and addresses my concerns raised in the discussion of the bug. What bug? Marius Gedminas -- Being really good at C++ is like being really good at using rocks to sharpen sticks. -- Thant Tessman signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] tracking back to what code caused a conflicting object change
A: Because it makes the logic of the discussion difficult to follow. Q: Why shouldn't I top post? A: No. Q: Should I top post? On Wed, Jul 28, 2010 at 04:19:29PM +0200, Adam GROSZER wrote: Hello Chris, I guess the best solution to find places which change something would be to hook into persistent.Persistent._p_changed, but that might not be an easy solution, 'cause it's all C. Set a breakpoint in ZODB.Connection.Connection.register, conditional on the object being of an instance of class TransientObject. CW PS: apologies for the inline forwarding, zope.org's MTAs are still CW incorrectly configured to reject messages forwarded as attachments... I've no problems with inline forwarding. I left the traceback here for context: CW Traceback (innermost last): CWModule Zope2.App.startup, line 173, in zpublisher_exception_hook CWModule ZPublisher.Publish, line 121, in publish CWModule Zope2.App.startup, line 240, in commit CWModule transaction._manager, line 96, in commit CWModule transaction._transaction, line 380, in commit CWModule transaction._transaction, line 378, in commit CWModule transaction._transaction, line 433, in _commitResources CWModule ZODB.Connection, line 479, in commit CWModule ZODB.Connection, line 1050, in _commit_savepoint CWModule tempstorage.TemporaryStorage, line 202, in store CW ConflictError: database conflict error (oid 0x046a, class CW Products.Transience.TransientObject.TransientObject, serial this txn CW started with 0x0387c9f23edea155 2010-07-27 10:26:14.735074, serial CW currently committed 0x0387c9f28a1ff766 2010-07-27 10:26:32.373016) CW To my knowledge, the request that throws this isn't touching the session CW or temp_folder, but I guess something is. CW How can I find out what code is actually touch this and so causing this CW conflict? Marius Gedminas -- 1 4m 5o 3l337! just got r00t on this a href=127.0.0.1k3wl site/a j00 sux0r5! signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Change in a mutable object(list) not being recognized by ZODB
On Sat, Apr 24, 2010 at 02:52:40PM +0530, rohit wrote:
I have a situation wherein a change made to an attribute(a list) of a ZODB
obj, is not being recognized. I have set the _p_changed = 1.
I do not see that in your code.
But still the ZODB doesnt recognize the change. On debugging i realised that
the obj has the change and then i do a transaction.commit() and close the
database. But on reaccessing the database i realise that the change hasnt
been registered.
Could someone please help me out on this one
You use a lot of nested non-persistent mutable objects (i.e. dicts).
Please be aware that if you modify any of them, you need to find the
containing *persistent* object and set its _p_changed to True.
Your life would be much easier if you used PersistentDict and
PersistentList instead of the builtin classes.
The code is
s=open_zodb()
root=s[0]
if not root.has_key(Clubs):
root[Clubs] = {}
ClubRoot=root[Clubs]
if not root.has_key(Managers):
root[Managers] = {}
ManagerRoot=root[Managers]
if not root.has_key(Players):
root[Players]={}
PlayerRoot=root[Players]
if not root.has_key(Count):
root[Count] = {}
CountRoot=root[Count]
club=findclubsbyname(ClubRoot, 'MI')
print club.Players
list=addplayers(PlayerRoot, CountRoot, b, 99, 100)
PlayerRoot=list[1]
CountRoot=list[0]
player=findplayerbyname(PlayerRoot, b)
player.Club=club
playerset=[]
playerset.append(player)
playerset.append(player)
#club.Players.append(player)
club.makeclubplayerassociation(playerset)
ClubRoot[club.ClubID]=club
root['Players']=PlayerRoot
root['Clubs']=ClubRoot
transaction.commit()
close_zodb(s)
//open_zodb() is a function which just opens connection to the databse.
findclubsbyname returns a club object .
addplayers is a function which adds a player and returns PlayerRoot and
CountRoot
findplayerbyname returns a player object
every club object has an attribte Players which is a list.
Marius Gedminas
--
No proper program contains an indication which as an operator-applied
occurrence identifies an operator-defining occurrence which as an
indication-applied occurrence identifies an indication-defining occurrence
different from the one identified by the given indication as an
indication-applied occurrence.
-- ALGOL 68 Report
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] DemoStorage stacking in zope.testing layers
On Mon, Apr 05, 2010 at 10:36:49PM +0800, Martin Aspeli wrote: I'm trying to do something like the following: - A test layer EmptyZODB sets up an empty ZODB using DemoStorage and exposes the DB object to other layers and tests. - A test layer Foo has EmptyZODB as a base. It opens a connection during layer setup, makes some changes, commits those, and closes the connection. - A test layer Bar has Foo as a base. It pushes a new DemoStorage on top of the one that was modified by Foo, opens a new connection, makes changes, commits those, and closes the connection. - On layer tear-down, Bar pops the demo storage, leaving the pristine fixture from Foo. I did this for SchoolTool once (poorly, because there were no layers yet). Note that when you stack multiple DemoStorages, ZODB gets very noticeably slow. As Hanno mentioned, zope.app.testing.functional also does this (with just one level of DemoStorage, substituting a new one for every test). You may want to looking up older versions of functional.py, since those were much simpler (because they didn't have support for multiple simultaneous databases). The problem with this scenario is that the database that was created by EmptyZODB and then modified by Foo is by necessity already active (even if it has no open connections). From my reading of the docs/interfaces, it's not possible to swap out a storage of a DB object, i.e. the storage is only settable in the constructor. What is the correct way to use DemoStorage stacking in test layers? Close and discard the old database object, create a new demo storage wrapping the old storage, create a new DB for it. Should Bar create a new DB object of its own and expose that to the tests that need it? Or is there some way to keep the same DB object (which would make the tests easier to write in this case) and simply tell it to use the top of the stack at all times? IMHO it's easier to find a way to pass the current database to the tests than to make the database switch storages on the fly. Marius Gedminas -- Never trust a smiling Gates. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Understanding the ZODB cache-size option
0.00 talinterpreter.py:582:do_setLocal_tal 0.00 2108.00 0.00 simpleviewclass.py:43:__call__ 0.00 2108.00 0.00 skin.py:119:render 0.00 2108.00 0.00 tales.py:54:__call__ 0.00 2108.00 0.00 talinterpreter.py:741:do_insertStructure_tal 0.00 2105.00 0.00 cal.py:655:dayEvents 0.00 2105.00 0.00 string:1:module 0.00 2105.00 0.00 cal.py:705:getDays 0.00 2105.00 0.00 cal.py:720:_getDays 0.00 2105.00 0.00 cal.py:683:getEvents 0.00 2105.00 0.00 cal.py:895:getDays 0.00 2105.00 0.00 pythonexpr.py:56:__call__ 0.00 2104.00 0.00 cal.py:673:getCalendars 0.00 2104.00 0.00 overlay.py:199:getCalendars 0.00114.00 0.00 __init__.py:647:collectTimetableSourceObjects 0.00114.00 0.00 source.py:48:getTimetableSourceObjects 0.00102.00 0.00 relationship.py:210:getRelatedObjects 0.00 65.00 0.00 hooks.py:104:adapter_hook 0.00 65.00 0.00 adapter.py:80:__call__ 0.00 22.00 0.00 registry.py:322:subscribers 0.00 22.00 0.00 _api.py:132:subscribers 0.00 22.00 0.00 annotatable.py:32:getRelationshipLinks 0.00 21.00 0.00 expressions.py:182:_eval 0.00 21.00 0.00 expressions.py:214:__call__ 0.00 17.00 0.00 talinterpreter.py:379:do_startTag 0.00 17.00 0.00 talinterpreter.py:468:attrAction_tal 0.00 17.00 0.00 talinterpreter.py:850:do_condition 0.00 17.00 0.00 tales.py:703:evaluateText 0.00 16.00 0.00 cal.py:381:linkAllowed 0.00 16.00 0.00 talinterpreter.py:818:do_loop_tal 0.00 10.00 0.00 policy.py:47:checkByAdaptation 0.00 10.00 0.00 crowds.py:225:contains 0.00 10.00 0.00 checker.py:88:canWrite 0.00 10.00 0.00 section.py:284:contains 0.00 10.00 0.00 policy.py:36:checkPermission 0.00 10.00 0.00 relationship.py:362:__iter__ 0.00 10.00 0.00 crowds.py:83:contains 0.00 6.00 0.00 absoluteurl.py:70:__call__ 0.00 6.00 0.00 absoluteurl.py:34:absoluteURL 0.00 6.00 0.00 absoluteurl.py:43:__str__ 0.00 6.00 0.00 cal.py:322:viewLink 0.00 4.00 0.00 registry.py:238:queryMultiAdapter 0.00 4.00 0.00 _api.py:114:queryMultiAdapter 0.00 3.00 0.00 _api.py:108:getMultiAdapter 0.00 1.00 0.00 engine.py:53:__call__ 0.00 1.00 0.00 expressions.py:108:_eval 0.00 1.00 0.00 adapters.py:96:traversePathElement 0.00 1.00 0.00 namespace.py:42:namespaceLookup 0.00 1.00 0.00 schoolyear.py:102:getActiveSchoolYear 0.00 1.00 0.00 mixins.py:102:expand 0.00 1.00 0.00 UserDict.py:57:get --- Sample count: 2126 Total time: 2126.00 seconds 2207427size 1950 class 'schooltool.timetable.model.PersistentTimetableCalendarEvent' 47 type 'BTrees.OOBTree.OOBTree' 27 class 'schooltool.relationship.relationship.Link' 23 class 'schooltool.relationship.relationship.LinkSet' 21 class 'persistent.mapping.PersistentMapping' 20 class 'schooltool.app.cal.Calendar' 20 class 'schooltool.course.section.Section' 7 class 'persistent.list.PersistentList' 2 class 'schooltool.schoolyear.schoolyear.SchoolYear' 2 class 'schooltool.group.group.Group' 1 type 'BTrees.IOBTree.IOBucket' 1 class 'schooltool.term.term.Term' 1 class 'schooltool.schoolyear.schoolyear.SchoolYearContainer' 1 type 'BTrees.IOBTree.IOBTree' 1 class 'zope.session.session.SessionPkgData' 1 class 'schooltool.course.section.SectionContainer' 1 class 'zope.intid.IntIds' Marius Gedminas -- Did you know that 7/5 people don't know how to use fractions? signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] free space for FileStorage, and possible issues with repozo (was: Missing Content)
On Sat, Jan 30, 2010 at 12:52:30PM -0500, Jim Fulton wrote: On Sat, Jan 30, 2010 at 6:18 AM, grahamperrin g.j.per...@bton.ac.uk wrote: Michael Havard wrote: We're missing 16 days worth of content. Python 2.3.5, Zope 2.8.4, Plone 2.1.2, and a host of other products. ZEO configuration. Persistent disk caching is on. Yesterday everything was fine. All the content was there and accessible. I recall reading about a case that involved quietly running out of free space on the volume used for FileStorage. Something like … the most recent content was in RAM — and the running ZEO cluster seemed to work fine with that (no adverse effect on the end user, at the time) — but that content was not committed to disk (and so, was probably unrecoverable following a restart of the OS). If you remove an open file storage (on a Unix-like system), the file still exists, but it is no longer accessible from the original directory. New transactions are committed to this still existing file, not ram. It is also possible to resurrect the deleted file as long as the process keeping it open doesn't quit: 1. find the pid and fd with lsof 2. cat /proc/$pid/fd/$fd /new/path (sadly, ln doesn't seem to work -- Invalid cross-device link -- so you need twice as much disk space) Marius Gedminas -- Only great masters of style can succeed in being obtuse. -- Oscar Wilde Most UNIX programmers are great masters of style. -- The Unnamed Usenetter signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Persistent object has empty __dict__ for a little while
On Wed, Jan 06, 2010 at 06:37:31PM -0500, Jim Fulton wrote: __dict__ is a special variable. Accessing it doesn't cause an object to be activated. Rather than doing: clone.__dict__.update(field.__dict__) I would do: clone.__setstate__(field.__getstate__) Nitpick: clone.__setstate__(field.__getstate__()) Alternatively, you can assure that field is acticated first, using _p_activate or by accessing an attribute. Regards, Marius Gedminas -- Those of you thinking that grown men with serious jobs should be above such phrases as way friendlier should realise that in the 21st century, adult appropriation of infantilisms is de rigueur, innit? -- Stuart Jeffries signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Undo alternatives
On Thu, Dec 24, 2009 at 09:37:22AM -0500, Jim Fulton wrote: As I mentioned in another message, undo is staying, but, as many also mentioned in that thread, undo often doesn't work due to conflicts. I thought I'd mention some alternatives: - Time travel ZODB has a number of facilities for time travel: - Object history There's an API for looking at changes over time. For simple objects, recovering from a misshap can often be as simple as copying state from an old revision. Zope 2 has (or had) a built in view for this. And zodbbrowser allows you to do this for Zope 3. Marius Gedminas -- System going down at 5 this afternoon to install scheduler bug. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Locating backrefs of persistent objects
I'd like to implement the following feature for zodbbrowser: given a persistent object X, find all other persistent objects that have references to it. My use case for this is debugging: e.g. given a log entry that mentions a conflict error about an OOBTreeBucket, I'd like to know which OOBTree object contains it, and what object wraps the OOBTree, and at that point I usually see a container with a __name__ and __parent__ and can figure out what's going on. I realize that ZODB doesn't keep track of backreferences, so the only way to get that is to do a full object traversal, which is expensive in both CPU and memory. Given all that, what's the best (most efficient) way of computing the backreferences either for a single object, or for _all_ objects in the ZODB (that I could then cache for later use)? * implement something by hand, by using storage iterators and extracting forward references from each pickle or * implement something by hand, by doing graph traversal (BFS/DFS) starting from the root object or * reuse existing object graph traversal routines of the ZODB garbage collector ? Is there existing code I could use for inspiration? zodbex? zc.zodbgc? Bits and pieces of ZODB itself? The wishlist bug for this feature is here: https://bugs.launchpad.net/zodbbrowser/+bug/497758 Marius Gedminas -- Linux don't need no steenkin' viruses. The users can destroy the system all by themselves -- Peter Dalgaard in comp.os.linux.misc signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 11:08:25AM +, Chris Withers wrote: Marius Gedminas wrote: * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 it's gone. Solved by peeking into the method signature with inspect.getargspec() and supplying a version only if it's needed. I believe this may be fixed with 3.9.4 is released. ...but it sounds like your code isn't ending up using the Z303 adapter, which is why you're running into the problem. What's Z303? How are you mixing versions? 3.8 on client and 3.9 on server or? I'm not mixing versions at runtime; my intent is to write code that works with either ZODB 3.8.x or 3.9.x. You can see it here in all its glory: http://bazaar.launchpad.net/~zodbbrowser-dev/zodbbrowser/trunk/annotate/head:/src/zodbbrowser/history.py#L30 http://bazaar.launchpad.net/~zodbbrowser-dev/zodbbrowser/trunk/annotate/head:/src/zodbbrowser/state.py#L115 (sorry, I lost the previous thread and saw this as a new one when Jim changed the subject line...) This is a new topic, completely unrelated to the previous thread. Marius Gedminas -- I'm a shareware signature! Send $2 if you use me, $10 for a manual. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 10:08:36AM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 5:58 AM, Marius Gedminas mar...@gedmin.as wrote: On Wed, Dec 09, 2009 at 01:04:03PM -0500, Jim Fulton wrote: On Wed, Dec 9, 2009 at 12:06 PM, Marius Gedminas mar...@gedmin.as wrote: ... (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very ugly hacks I managed.) This sounds like something that needs to be fixed. Can you share some of the issues you ran into? (Or maybe file bugs reports.) Two issues only: * In ZODB 3.8 PersistentDict and PersistentMapping are unrelated classes, so if I want to have adapters for both, I have to register them separately in ZCML. In ZODB 3.9, PersistentDict is an alias for PersistentMapping, so I get a ZCML configuration conflict error if I have adapter directives for both. OK, it's impractical to do anything about this. Solved by defining a decoy class for one of the adapters so they don't conflict. Hm, I'm not sure I follow this. Code: if PersistentMapping is PersistentDict: # ...snip large comment explaining this... class DecoyPersistentDict(PersistentMapping): Decoy to avoid ZCML errors while supporting both ZODB 3.8 and 3.9. class PersistentDictState(PersistentMappingState): Decoy to avoid ZCML errors while supporting both ZODB 3.8 and 3.9. adapts(DecoyPersistentDict, dict, None) else: class PersistentDictState(PersistentMappingState): Convenient access to a persistent dict's items. adapts(PersistentDict, dict, None) ZCML: adapter factory=.state.PersistentDictState / adapter factory=.state.PersistentMappingState / You could have used a different approach of providing the second adapter registration as an override. Oooh, neat hack. But I'm not sure I like it better than the one I implemented. * In ZODB 3.8, the 'version' argument of ClientStorage.history (as well as other kinds of storages, I suppose) is mandatory. In ZODB 3.9 it's gone. I went back and corrected my statement (originally it referred to FileStorage.history, and I was thinking about ClientStorage in my parenthetical statement), and then I of course forgot to remove the now-obsolete parenthetical statement itself. It's mandatory in ClientStorage and optional in FileStorage. (MappingStorage and DemoStorage don't have a history method in 3.8. I suggest we treat this as a bug in 3.8 and make the version argument to ClientStorage's history method optional in a 3.8 bug fix release. +1 Solved by peeking into the method signature with inspect.getargspec() and supplying a version only if it's needed. Ick. :) :) In particular, I think it should be a goal that it isn't too hard to write code that works with both ZODB 3.8 and 3.9. Normal code doesn't usually play with storage.history, I'd say, so I'm not too peeved by having to do various contortions. Zope 2 had (has?) a standard view for looking at an object's history that I found very useful. I wish there was one for Zope3. Well, there is IStorage, and I think all storages (except for FileStorage) implemented it faithfully (oid, version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated the interface by renaming size=1 to length=1, but it's fixed in 3.9. I personally would like to be able to have size=None imply unlimited, give me as much as you've got), but works just as well in practice. What I miss _a lot_ is a way to look at the last several transactions. There's the iterator API, but it goes from oldest-to-latest, and when you've got a large DB, that takes a while (85 seconds for a Data.fs with 922737 transactions and 8 GB of data, with 100 MB/s read throughput). History can be very useful in my experience. Oh yes. This is why I stopped packing my production Data.fs'es and first started using storage.history() with a couple of nice wrappers from debugzope shell, and, later, convinced a coworker to write ZODBBrowser. Marius Gedminas -- I used to think I was indecisive, but now I'm not so sure. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas mar...@gedmin.as wrote: ... Well, there is IStorage, and I think all storages (except for FileStorage) implemented it faithfully I guess that depends on what you mean by faithfully. MappingStorage and Demostorage in 3.8 inherited a null implementation from BaseStorage and it's arguable how faithful that implementation is. (oid, version-that-got-removed-in-3.9, size=1). FileStorage in 3.8 violated the interface by renaming size=1 to length=1, but it's fixed in 3.9. and in 3.8. *checks* FileStorage in 3.8.1 violated the interface, to be precise. ZODB 3.8.1 is mandated by http://download.zope.org/zope3.4/3.4.0/versions.cfg, which our production app currently uses. (Kinda offtopic: Is there a newer 3.4 KGS release? http://download.zope.org/ tells me Welcome and nothing more. http://download.zope.org/zope3.4/ tells me in great big joyful letters at the top of the page zope 3.4.0 Has Been Released!) What I miss _a lot_ is a way to look at the last several transactions. There's the iterator API, but it goes from oldest-to-latest, and when you've got a large DB, that takes a while (85 seconds for a Data.fs with 922737 transactions and 8 GB of data, with 100 MB/s read throughput). In 3.9 for FileStorage, if you give a starting tid that is toward the end of the file, it will scan backward, saving a lot of time. Ah, but how can I get a valid tid that is toward the end of the file? I assume that if I give a tid that doesn't exist, I'll get an error -- which is the behavior oldstate() exhibits -- rather than the next existing transaction after that timestamp, or I could fudge by getting the current timestamp and subtracting 24 hours. I've attached an analysis script that illustrates iterating backward. Thank you! (Mutt is inconvenient: there's no way to look at attachments while you're replying to a message.) Someday, someone should get around to adding an option to the file-storage iterator to go backward. Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB 3.9/3.8 incompatibility (was Re: Data.fs size grows non-stop)
On Thu, Dec 10, 2009 at 02:07:06PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 1:50 PM, Marius Gedminas mar...@gedmin.as wrote: On Thu, Dec 10, 2009 at 12:41:11PM -0500, Jim Fulton wrote: On Thu, Dec 10, 2009 at 11:54 AM, Marius Gedminas mar...@gedmin.as wrote: In 3.9 for FileStorage, if you give a starting tid that is toward the end of the file, it will scan backward, saving a lot of time. Ah, but how can I get a valid tid that is toward the end of the file? I assume that if I give a tid that doesn't exist, I'll get an error -- which is the behavior oldstate() exhibits -- rather than the next existing transaction after that timestamp, or I could fudge by getting the current timestamp and subtracting 24 hours. No, the iterator starts at the first transactions who's tid is = the start tid. So, guess a time and use repr(ZODB.TimeStamp.TimeStamp(...)) Cool! Thank you, Marius Gedminas -- This message has been ROT-13 encrypted twice for higher security. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Data.fs size grows non-stop
On Wed, Dec 09, 2009 at 02:42:58PM +0100, Pedro Ferreira wrote:
Hello,
Just zodbbrowser with no prefix:
http://pypi.python.org/pypi/zodbbrowser
https://launchpad.net/zodbbrowser
It's a web-app: it can connect to your ZEO server so you can inspect the
DB while it's being used.
We tried this, but we currently get an error related with the general
security policy for zope.app. Maybe we need to install Zope?
No, it should be self-contained.
I'm usually testing with the Zope 3.4 KGS. Easiest way to do that is to
check out the sources and use buildout:
bzr get lp:zodbbrowser
cd zodbbrowser
python bootstrap.py
bin/buildout
bin/zodbbrowser --zeo localhost:1234
I'll test it with the latest zope.* packages from PyPI and see if I can
reproduce the error. Feel free to report a bug (and attach the
traceback you get) here: https://bugs.launchpad.net/zodbbrowser
This would be a very handy tool.
I'd suggest dumping the last few transactions with one of the ZODB
scripts (fsdump.py perhaps) and seeing what objects get modified.
That's what we've being doing, and we got some clues. We've modified
Jim's script in order to find out which OIDs are being rewritten, and
how much space they are taking, and this is a fragment of it:
OID class_name total_size percent_size n_pickles min_size avg_size max_size
'\x00\x00\x00\x00%T\x89{' BTrees.OOBTree.OOBucket 17402831841 30% 8683
1977885 2004241 2026518
'\x00\x00\x00\x00%T\x89|' BTrees.OOBTree.OOBucket 14204430890 24% 8683
1616904 1635889 1651956
'\x00\x00\x00\x00\x04dUH' MaKaC.common.indexes.StatusIndex 11955954522
20% 28513 418230 419315 420294
'\x00\x00\x00\x00%\xa0%\x7f' BTrees.OOBTree.OOBucket 3532998238 6% 11238
307112 314379 320647
'\x00\x00\x00\x00%\xa0%\x80' BTrees.OOBTree.OOBucket 2193843302 3% 11238
190816 195216 199007
'\x00\x00\x00\x00\x04\x8e\xb6\x04' BTrees.OOBTree.OOBucket 1728216003 3%
1953 880615 884903 887285
[...]
As you can see, we have an OOBucket occupying more than 2MB (!) per
write.
Ouch.
That's almost 17GB only considering the last 1M transactions of
the DB (we get ~3M transactions per week). We believe this bucket
belongs to some OOBTree-based index that we are using, whose values are
Python lists (maybe that was a bad choice to start with?). In any case,
how do OOBuckets work? Is it a simple key space segmentation strategy,
or are the values taken into account as well?
Our theory is that an OOBTree simply divides the N keys in K buckets,
and doesn't care about the contents. So, since we are adding very large
lists as values, the tree remains unbalanced, and since new contents
will be added to this last bucket, each rewrite will imply the addition
of ~2MB to the file storage.
Will the replacement of these lists with a persistent structure such as
a PersistentList solve the issue?
It should definitely help. Now, if modify one of the lists in a bucket,
it needs to append the whole 2 megs of data to the Data.fs. If you'd
used a PersistentList, it would only need to append as much data as your
persistent list contains. Assuming there are ~30 lists in a bucket,
you'd get 30x space savings.
Are those lists long? Are they modified often? What do they contain?
I'm sure you can get better space efficiency by redesigning the data
structure.
Marius Gedminas
--
Dijkstra probably hates me
-- Linus Torvalds, in kernel/sched.c
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Data.fs size grows non-stop
On Wed, Dec 09, 2009 at 06:09:36PM +0200, Marius Gedminas wrote: On Wed, Dec 09, 2009 at 02:42:58PM +0100, Pedro Ferreira wrote: Hello, Just zodbbrowser with no prefix: http://pypi.python.org/pypi/zodbbrowser https://launchpad.net/zodbbrowser It's a web-app: it can connect to your ZEO server so you can inspect the DB while it's being used. We tried this, but we currently get an error related with the general security policy for zope.app. Maybe we need to install Zope? No, it should be self-contained. I'm usually testing with the Zope 3.4 KGS. I've just released zodbbrowser 0.6.1 which works with ZODB 3.9.x and latest versions of other packages too. (Supporting both ZODB 3.8 and 3.9 is kinda tricky, but with some very ugly hacks I managed.) Marius Gedminas -- Any sufficiently advanced Operating System is indistinguishable from Linux. -- Jim Dennis signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Data.fs size grows non-stop
On Mon, Dec 07, 2009 at 10:58:20PM +0100, Roger wrote: what I whould todo as next, try z3c.zodbbrowser or the other ZODB introspector tool from mgedmin, don't remember the name of the tool. Just zodbbrowser with no prefix: http://pypi.python.org/pypi/zodbbrowser https://launchpad.net/zodbbrowser It's a web-app: it can connect to your ZEO server so you can inspect the DB while it's being used. I'd suggest dumping the last few transactions with one of the ZODB scripts (fsdump.py perhaps) and seeing what objects get modified. If you know an OID of an object, you can look at it with ZODBBrowser by entering it in the URL, e.g. @@zodbbrowser?oid=0x1234. Seeing the object might give some clue where it comes from. Or, if you see a long history list for an object, you'll see what URLs were accessed during the requests that made modifications here. Sadly, currently ZODBBrowser doesn't have the ability to look for backreferences (planned in my mental TODO list---but it'll be a very expensive operation, especially on a large DB, since ZODB itself doesn't keep track), and it doesn't let you view transactions (also on my mental TODO list, also a bit expensive). the z3c.zoodbbrowser can open a ZODB with a pythin GUI. There you can iterate over each objects in the ZODB. You will probably see more then you see in the UI, especialy if some objects get created which you don't expect. Marius Gedminas -- There's a special hell 4 people who replace words with numbers. -- Ben Yahtzee Croshaw signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Wed, May 27, 2009 at 12:17:36PM +0200, Hanno Schlichting wrote: Chris Withers wrote: Hanno Schlichting wrote: They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call __init__ and I don't see why the DWIM magic would be needed anyway, since everything has already been parsed. How would a new instance of a class be constructed without calling the init or new? You're cleverly changing the question ;-) *Most* objects are unpickled by calling __new__ followed by __setstate__, but without calling __init__. Chris is therefore understandably surprised. Old-style classes that define __getinitargs__ will get their __init__ called during unpickling, though, and DateTime is such a class. You could also cause the __init__ to be called by defining __reduce__ or __reduce_ex__ appropriately. This is all described in great detail at http://python.org/doc/current/library/pickle.html#pickle-protocol Look at the _instantiate method in pickle.py, when it does: value = klass(*args) What happens on unpickling is that a new DateTime instance representing just now is generated and then that instance is updated with the values from the pickle. Marius Gedminas -- Every nonempty totally-disconnected perfect compact metric space is homeomorphic to the Cantor set. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Wed, May 27, 2009 at 03:35:26PM +0200, Hanno Schlichting wrote: Marius Gedminas wrote: On Wed, May 27, 2009 at 12:17:36PM +0200, Hanno Schlichting wrote: Chris Withers wrote: Hanno Schlichting wrote: They are incredibly expensive to unpickle since all the DWIM magic in their __init__ get called each time, though. How come? Unpickling doesn't call __init__ and I don't see why the DWIM magic would be needed anyway, since everything has already been parsed. How would a new instance of a class be constructed without calling the init or new? You're cleverly changing the question ;-) *Most* objects are unpickled by calling __new__ followed by __setstate__, but without calling __init__. Chris is therefore understandably surprised. Hhm, no. From what I read most objects are created by the following: class _EmptyClass: pass value = _EmptyClass() value.__class__ = klass The __new__ is only called when your new-style class has a __getnewargs__ method, which none of the standard types have. And even then it would only be used for pickle protocol 2, but the ZODB uses protocol 1. It's an Internet Law that every correction must have at least one mistake. Mine was assuming that __new__ is always invoked. Old-style classes that define __getinitargs__ will get their __init__ called during unpickling, though, and DateTime is such a class. Ah, ok. I missed that this only happens in conjunction with __getinitargs__. Marius Gedminas -- Bumper sticker: No radio - Already stolen. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Tue, May 26, 2009 at 12:00:51PM +0200, Pedro Ferreira wrote:
Jim Fulton wrote:
What was in the ZEO server log when this happened?
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45138):
ManagedServerConnection ('137.138.128.213', 45138)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.213', 45139):
ManagedServerConnection ('137.138.128.213', 45139)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59721):
ManagedServerConnection ('137.138.128.212', 59721)
2009-05-24T12:22:54 (28965) new connection ('137.138.128.212', 59722):
ManagedServerConnection ('137.138.128.212', 59722)
2009-05-24T12:22:54 (137.138.128.213:45138) received handshake 'Z303'
2009-05-24T12:22:54 (28965) Error raised in delayed method
Traceback (most recent call last):
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
1009, in run
result = self._method(*self._args)
File /usr/lib64/python2.3/site-packages/ZEO/StorageServer.py, line
345, in _pack_impl
self.storage.pack(time, referencesf)
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 1372, in pack
self._save_index()
File
/usr/lib64/python2.3/site-packages/ZODB/FileStorage/FileStorage.py,
line 252, in _save_index
p.dump(info)
RuntimeError: maximum recursion depth exceeded
That's shorter than I expected.
OTOH the `p` here probably refers to a cPickle.Pickler object, and I've
never seen recursion depth errors caused by C code before. It makes
sense.
Also, how many objects are in your database?
Hmmm... I have no idea... is there an easy way of calculating that?
There's a script called analyze.py that comes with ZODB. It will give
you summaries grouped by object class of the counts and sizes of all
your persistent objects.
Marius Gedminas
--
The only way to learn a new programming language is by writing programs in it.
-- Brian Kernighan
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Tue, May 26, 2009 at 04:16:29PM +0200, Pedro Ferreira wrote: Also, is there any documentation about the basic structures of the database available? We found some information spread through different sites, but we couldn't find exhaustive documentation for the API (information about the different kinds of persistent classes, etc...). Is there any documentation on this? No, but you have something even better than documentation -- the source! More seriously, when I needed it, I've been able to get enough information about the FileStorage internal structure from code comments and docstrings, and some jumping around following the flow of functions. Marius Gedminas -- I read I forget; I see I remember; I teach I understand. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN
On Mon, May 25, 2009 at 01:44:46PM +0200, Pedro Ferreira wrote: We're using ZODB for the Indico Project (at CERN), since 2004, without any kind of problem. However, today, our database went down and we can't find a way to recover it. This is a major issue, since we have ~4000 users depending on this application, and we're simply not able to access the data in any way. Ouch. Around 00:30 tonight the database went down, and since, all the connections are refused. This means that you're using ZEO, right? Have you tried to use strace to see what it's doing? Is it using any CPU time? We tried to restart the database, but the script seems to hang, while trying to create the index: -rw-r--r-- 1 root root 6396734704 May 25 13:21 dataout.fs -rw-r--r-- 1 root root 173 May 25 12:21 dataout.fs.index -rw-r--r-- 1 root root 229755165 May 25 13:22 dataout.fs.index.index_tmp -rw-r--r-- 1 root root 7 May 25 12:21 dataout.fs.lock -rw-r--r-- 1 root root70956364 May 25 13:21 dataout.fs.tmp We tried to do fsrecovery, but it says 0 bytes removed during recovery, and the result ends up being the same. We tried it in different machines, with no success. In one of them, after a while trying to create the index, a Python exception was thrown, saying maximum recursion depth exceeded. I'm not intimately familiar with the internals of ZODB. If it's doing object graph traversals recursively, and if your object graph is very deep, you may mitigate this by calling, e.g. sys.setrecursionlimit(2 * sys.getrecursionlimit()) We're using 3.4 in production, but we've tried with 3.9 and 3.9b0 as well, with no success. We're getting kind of desperate, since the same seems to happen with yesterday's backup, and trying to restore previous backups with repozo raises a CRC Error. Has anyone ever experienced this? Any clues on how to solve this problem? We'd really appreciate you could help us out, since this is becoming a big issue here at CERN (a lot of people's work depends on this). Good luck! Marius Gedminas -- Linux: The Ultimate NT Service Pack signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Estrange behaviour with BTreeFolder2
On Wed, Apr 01, 2009 at 11:46:01AM +0200, Santi Camps wrote:
I'm having memory problems and, after some debugging with gc, I see
some estrange behaviours with BTreeFolder2 (or with BTrees itself, I'm
not sure). I've a BTreeFolder2 with 4 objects, more o less.
Some of them are emails. Just accessing the container, about 700
mails are loaded in memory !!
from AccessControl.SecurityManagement import newSecurityManager
from Testing.makerequest import makerequest
user = app.beta.acl_users.getUser('manager').__of__(app.beta.acl_users)
newSecurityManager({}, user)
app = makerequest(app)
km = app.beta
km.portal_repository._tree
import gc
objects = gc.get_objects()
objects2 = [obj for obj in objects if getattr(obj, '__class__', None)]
(I'm a bit surprised that you get objects without a __class__ attribute
-- could you elaborate about those? Old-style class instances?
Extension types?)
emails = [obj for obj in objects2 if 'mail' in obj.__class__.__name__.lower()]
How many different classes are there with 'mail' in the (lowercased)
class name?
Do they all inherit from Persistent?
print len(emails)
697
Following gc references, I see this mails are referenced from a
OOBucket, that is referenced by another OOBucket, ...
I assume those mails are values rather than keys.
and some of them
are referenced from persistent.PickleCache (instead I've configured
cache-size = 0 in zope.conf). Seems that some buckets and its
referenced objects are readed during btree loading. So, the garbage
collector never clears this from memory, and after some hours of
intensive work, the server RAM is finished
Is that normal ? I'm making some mistake ? Anybody knows any way to solve
it ?
Your description would make perfect sense to me if your mail classes
weren't subclasses of Persistent (and the fix would be: make them
subclass Persistent).
Marius Gedminas
--
I used to (somewhat) sneer at people who describe themselves thus: I program
HTML. Then I tried to make a web-site look as I wanted it to (ie. not
grotesquely ugly), wow - perhaps they were onto something. Some think there is
a pile of broken mess in our desktop, but at least you know where you are with
a GtkHBox.
-- Michael Meeks
signature.asc
Description: Digital signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] svn.zope.org up again?
On Tue, Nov 11, 2008 at 08:51:28AM +0100, Adam GROSZER wrote: Hello Stephan, svn: Can't open file '/svn/repos/main/db/revs/70320': No such file or directory program finished with exit code 1 Seems that one revision did not get restored. It breaks also on show log, where other folders with older revisions seem to work. I've a full svn mirror up to rev 92837 if it turns out to be needed. Marius Gedminas -- IBM motto: If you can't read our assembly language, you must be borderline dyslexic, and we don't want you to mess with it anyway -- Linus Torvalds signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with broken
On Wed, Nov 05, 2008 at 03:59:07PM +0100, Christian Theune wrote: On Wed, 2008-11-05 at 16:00 +0200, Marius Gedminas wrote: On Wed, Nov 05, 2008 at 11:48:35AM +0100, Christian Theune wrote: On Wed, 2008-11-05 at 11:31 +0100, Laurence Rowe wrote: Broken objects occur when the class for a pickled object cannot be imported. To change the location of a class, you need to provide an alias at the old location so that the object can be unpickled, i.e. MyOldClassName = MyNewClassName. You can only remove MyOldClassName after you have updated all of the pickles (with your code below). Note: We made a relatively good experience by removing those symbols from their original modules and provide a generation which first sets up dummies, then runs migration code to remove/rebase those objects and then remove the dummies again. How do you do that? Monkey-patching? Yes. Monkey-patching *and* monkey-unpatching. Thought so. By the way, this has a potential failure mode when you need to evolve more than one generation, and a previous generation script needs to access those objects in the old location before your new generation script sets up the dummies. Marius Gedminas -- What can I do with Python that I can't do with C#? You can go home on time at the end of the day. -- Daniel Klein signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with broken
On Wed, Nov 05, 2008 at 01:15:58PM +0100, Adam GROSZER wrote: Thanks for the advices on the generation script, but I think the problem lies in that the broken object is loaded fine but cannot be written back again to ZODB. Yes. You can't do anything with broken objects. They're there only so that you don't get an exception when loading an object that refers to a broken object. Or do I misunderstand that broken is a sort of proxy for the missing class that allows it to be around? Yes, but being around is a rather narrow goal that doesn't allow you to do anything with it. Marius Gedminas -- Power corrupts, but we need electricity. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.8.1
On Fri, Oct 10, 2008 at 10:43:57AM -0700, Marius Gedminas wrote: In the mean time I updated the buildbot config to run the tests separately. I don't know offhand if buildbot 0.7.4 supports the interlocks documented in 0.7.9's documentation[1], so I just created a new scheduler that runs the other set of tests 2 hours later. [1] http://buildbot.net/repos/release/docs/buildbot.html#Interlocks It seems that one ZEO test consistently (7 times in a row) fails on the 32-bit build slave, on both Python versions, even when run in isolation: ZODB3-3.8.1b9-py2.5-linux-i686.egg/ZEO/tests/invalidations_while_connecting.test Marius Gedminas -- The First and Second Rules of Program Optimisation: 1. Don't do it. 2. (For experts only!): Don't do it yet. -- Michael A Jackson signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] 3.8.1
On Fri, Oct 10, 2008 at 10:20:51AM -0400, Jim Fulton wrote: On Oct 10, 2008, at 5:14 AM, Christophe Combelles wrote: http://zope3.pov.lt/buildbot/ There is a new failure on ZEO/tests/ invalidations_while_connecting.test An there is a problem with file locking on the 64bit slaves with: checkMultipleServers checkNoVerificationOnServerRestartWith2Clients checkReconnectReadOnly checkReconnectSwitch testNOK checkDisconnectedAbort checkMultipleAddresses Christophe The tests pass if they are run alone : http://zope3.pov.lt/buildbot/ It would be good to fix these concurrency problems In the mean time I updated the buildbot config to run the tests separately. I don't know offhand if buildbot 0.7.4 supports the interlocks documented in 0.7.9's documentation[1], so I just created a new scheduler that runs the other set of tests 2 hours later. [1] http://buildbot.net/repos/release/docs/buildbot.html#Interlocks Yeah. ZEO and ZODB (maybe just ZEO) tests are often sloppy about the files they create. I recently modified the buildout to create temporary files in the test part directory, which should have helped. One of my near future projects will be to try to address this problem. Speaking of temporary files, running the tests leaves over a 100 new temporary files and directories in /tmp every night. I had to create a cron script to clean them up to avoid completely filling up the disk. Marius Gedminas -- When we say we want readable code, we don't mean we want to sit in a comfortable chair and page through a Java-saga. --- http://www.wordaligned.org/articles/pitching-python-in-three-syllables signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Broken instances after refactoring in ZODB
On Fri, Oct 03, 2008 at 11:37:35AM +0100, Adam wrote: Hi, I wonder if anyone can help me. I've got a custom Python cgi which runs in one directory using ZODB for the database. It runs in a directory off the site root ie. /System It's got a bit unwieldy in there so I wanted to separate the core modules out as I don't really touch them any more, so adding subdirectories: /System/Core /System/Application In System, I created a dummy frontend CGI like the following: # /System/frontend.py import ZODB from Core import * from Application import * site = Start.ProcessRequest() # Start is a module in Core site.Main() For some reason, all the Persistent derived objects in the database are returning: broken Module.Class instance Does ZODB do something with __import__ modules which may cause it to lose the references to the classes/modules? The ZODB stores fully qualified class names (package.module.class). If you move the modules around, ZODB can no longer find the classes in their old locations. Is there any way to fix the database Yes, but it's not pretty: 1. create modules in their old locations that import all the classes from the new locations (and do nothing else). 2. write a script that opens the DB, iterates over all persistent objects in it, touches every single one of them (obj._p_activate() followed by obj._p_changed = True) and commits the transaction. If you've done everything right, now the database should refer to all the classes using their new locations. or do I have to return to putting all my .py files in one directory. Third alternative: create modules for backwards compatibility (just the step 1 described above). Marius Gedminas -- Those who can't write, write manuals. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Very large amounts of output from fsrefs.py
On Wed, Aug 27, 2008 at 01:32:53PM +0200, Ingvar Hagelund wrote: We have an solution based on a third party zope based framework, with 12 Zope instances accessing ZODB via ZEO. ... Now, writing to the ZEO database is very slow, and the same goes for searching, which is a problem for the users that put information into the solution. ... We want to sort out if there are any obvious data points on tuning ZEO that we have missed, or that there are errors or corruption in the database. ... Software environment: ZEO from Zope 2.10.3, python-2.4.4, Debian GNU/Linux, Data.fs on ext3 filesystem ZEO had a bug, fixed in rev 73871, where it would spend up to 10ms (your OSes scheduling timeslice) in idle sleep for every object received from the ZEO server, resulting in very long request processing times on an otherwise idle server, if a request had to load hundreds or thousands of persistent objects. I do not remember the mapping from Zope 2.x versions to ZODB/ZEO versions offhand (nor do I remember how to look that up). It may be worth checking if this applies to you. Average cpu usage on the box running ZEO is some 10%, and never above 65% (of one cpu core). This is what makes me suspect you may be seeing that bug. Running fsrefs.py on Data.fs gives 1507079 instances of refers to invalid objects. This might indicate a problem, I would guess. This is beyond my expertise. I'd be worried too. Marius Gedminas -- Just a note: quantum gravity discussions are _clearly_ on-topic in c.l.py. -- Peter Hansen on comp.lang.python signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB fails on test
On Sun, Aug 03, 2008 at 11:36:41AM +0300, Malka Cymbalista wrote: We are trying to install ZODB3-3.4.2 on a Linux machine running Red Hat Enterprise Linux AS release 4 and python 2.5.2. ... We then ran python test.py and got the results that are in the attached file. This does not look right. Can we continue with the installation? Any help will be appreciated. ... [EMAIL PROTECTED] ZODB3-3.4.2]$ python test.py Running tests from build/lib.linux-x86_64-2.5 Traceback (most recent call last): File test.py, line 61, in module testrunner.run(defaults) File build/lib.linux-x86_64-2.5/zope/testing/testrunner.py, line 149, in run options = get_options(args, defaults) File build/lib.linux-x86_64-2.5/zope/testing/testrunner.py, line 1269, in get_options default_setup, _ = parser.parse_args(default_setup_args) File /usr/local/lib/python2.5/optparse.py, line 1385, in parse_args stop = self._process_args(largs, rargs, values) File /usr/local/lib/python2.5/optparse.py, line 1425, in _process_args self._process_long_opt(rargs, values) File /usr/local/lib/python2.5/optparse.py, line 1500, in _process_long_opt option.process(opt, value, values, self) File /usr/local/lib/python2.5/optparse.py, line 776, in process value = self.convert_value(opt, value) File /usr/local/lib/python2.5/optparse.py, line 768, in convert_value return self.check_value(opt, value) File /usr/local/lib/python2.5/optparse.py, line 763, in check_value return checker(self, opt, value) File /usr/local/lib/python2.5/optparse.py, line 420, in check_builtin return cvt(value) File /usr/local/lib/python2.5/optparse.py, line 407, in _parse_int return _parse_num(val, int) File /usr/local/lib/python2.5/optparse.py, line 394, in _parse_num if val[:2].lower() == 0x: # hexadecimal TypeError: 'int' object is unsubscriptable This is a rather interesting failure. It is caused by zope.testing.testrunner used in ZODB 3.4.2 assuming optparse accepts command line arguments that aren't strings. See http://zope3.pov.lt/trac/browser/zope.testing/tags/ZODB3.4.2/src/zope/testing/testrunner.py#L1257 You could fix that in your source tree by changing '--at-level', 1, to '--at-level', '1', in src/zope/testing/testrunner.py in your ZODB source tree and then rerunning setup.py build. Or you could switch to Python 2.3 or 2.4. Trying to use 3-year-old software (ZODB 3.4.2) in an unsupported configuration (Python 2.5) may have downsides. Marius Gedminas -- Anything can be made to work if you fiddle with it long enough. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Locking
On Wed, Jul 02, 2008 at 04:45:53PM -0400, Joseph Turian wrote: How do I lock zodb objects? I want the following process: See if key is present in root. [it's not] Lock key write-access, so that no other client will try to write key. Compute value for key. Write value for key. Unlock key. Usually you shouldn't need to do this. ZODB will detect conflicting write transactions and raise ConflictError that you're supposed to catch and retry up to a certain reasonable number of times (3 or 5). Marius Gedminas -- Key emulation: [ ] Intuitive [*] Emacs(Seen in an MCEdit dialog) signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Mutating class of an instance
(Adding zodb-dev@zope.org back to Cc:) On Thu, Jul 03, 2008 at 09:14:27AM -0400, Erik Dahl wrote: On Jul 2, 2008, at 10:55 AM, Marius Gedminas wrote: inst.__class__ = NewClass Is inst a persistent object? Then your only chance to get this to work is to find *all* objects that reference inst and mark them _p_changed. Yes inst is a persistent object. Hum ok I will give this a try. The class is also stored in every reference to that object. Well that something new to me... how come? Seems like they only need the oid. My guess is that's for performance reasons. When you load the state of an object X with an attribute 'y' referring to object Y, you end up with two objects in memory: X with all of its state, and Y which is a ghost (it only knows its own class and oid). To create the ghost you need to know its class. To avoid loading the pickle of Y when you only need X, every reference to Y stores both its oid and its class. If you look at ZODB/serialize.py and look at ObjectWriter.persistent_id, you'll see it stores references to persistent objects as (oid, class) tuples. Marius Gedminas -- find /lib/modules/2.4.17-expt/kernel/ -type f|while read i; do insmod $i; done You're sick. I like you. -- Andrew Morton on lkml signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB design. Size and saving records
On Sun, Jun 22, 2008 at 08:54:14AM -0700, tsmiller wrote:
ZODB guys,
I have a bookstore application in which I use the ZODB in a simple way. The
database saves stores and records in the stores.
I open the database as follows where the databasePath argument points to my
bookserver.fs file.
from ZODB import FileStorage, DB
import transaction
import BTrees.OOBTree, BTrees.IOBTree
def open(self, databasePath):
self.bookDatabase_storage =
FileStorage.FileStorage(databasePath)
Ouch! PEP-8 is not rocket science and it makes reading your code a more
pleasant experience.
self.bookDatabase_db =
DB(self.bookDatabase_storage)
self.bookDatabase_connection = self.bookDatabase_db.open()
self.dbRoot = self.bookDatabase_connection.root()
# our btrees
ibTree = BTrees.IOBTree
I define the key 'books' to be an IOBTree
db.dbRoot['books'] = ibTree.IOBTree()
And when the user creates a store I define the book number to be an IOBTree
db.dbRoot['books'][storeNumber] = ibTree.IOBTree()
Then I save the book using storeNumber and bookNumber as keys.
record = {'title':The Grapes of Wrath, 'author':John
Steinbeck,'publisher':Randomhouse}
Better make that
record = PersistentDict({'title': ...})
Also: mixing tabs and spaces is a bad idea.
db.dbRoot['books'][storeNumber][bookNumber] = record
So now I can qualify an entire store by using
currentStore = db.dbRoot['books'][storeNumber]
And a single book by
currentBook=
db.dbRoot['books'][storeNumber][bookNumber]
And look at the books in the store by
for k, v in currentStore.values():
I assume you mean 'items()' rather than 'values()'.
print k, v
I have two questions.
1) When I already have a storeNumber and I save a record to
db.dbRoot['books'][storeNumber][bookNumber] =
record
I have to set the _p_changed flag on the 'books' IOBTREE
structure to get
the book to save.
db.dbRoot['books']._p_changed = True
No. Because you use IOBTree objects, they notice item additions and
removals automatically. Changes too, if you do things like
btree[existing_key] = new_item
but not changes of stored items, for example
btree[existing_key].attribute = new_value
If you store only persisten objects (such as PersistentDict, or custom
classes inheriting from Persistent) and immutable objects (such as
strings and ints) in your database, you will never have to worry about
setting _p_changed.
Which means that it saves the ENTIRE 'books' IOBTREE' structure
every time
a save a single book.
It should save the top-most 'books' IOBTree (pointlessly, because it
wasn't actually changed) and the store IOBTree only, and none of the
other store BTrees. Also, I believe it should only store those BTree
buckets that actually changed. But note that since you use
nonpersistent dicts, the store BTrees will have to repeatedly save them
every time you change a book in that store BTree (at least the ones in
the affected BTree bucket).
(at least it looks like it is doing this). When I
edit a book and save it the database grows by more than 64k. And it looks
like it will get worse and worse as more books are added.
Change the records to be PersistentDicts and you'll see space savings.
Am I looking at this correctly. Or am I doing something really
ignorant?
Yes: storing nonpersistent objects.
2) When I assign a value to the db such as
db.dbRoot['books'][storeNumber][bookNumber] = record
I initially assumed that setting the _p_changed flag on
the storeNumber
key would only save the single record that I want to save. As mentioned
above, I am setting the flag on the 'books' IOBTREE also. Should I have to
set it on both. I have come to the conclusion that the _p_changed flag must
be set at the highest key level. ie.. 'books'. Again, Am I doing something
really ignorant?
The _p_changed should be set on the nearest enclosing persistent object,
so with the data structure you have, you'd only need to set it on
db.dbRoot['books'][storeNumber]._p_changed = True
whenever you changed a value inside a book record
db.dbRoot['books'][storeNumber][bookNumber]['author'] = 'Someone'
I'm only telling you this for illustration; the recommended way is to
never store nonpersistent mutable objects in the database.
Thank you for your responses. I really need to know and get this fixed
before my wife divorces me! She spends time entering books and we still
seem to not really know when the changes are going to permanently be saved.
Ouch.
Marius
Re: [ZODB-Dev] Mutating class of an instance
On Wed, Jul 02, 2008 at 08:07:56AM -0400, Erik Dahl wrote: I have a situation where I want to mutate the class of an instance like this: inst.__class__ = NewClass Is inst a persistent object? Then your only chance to get this to work is to find *all* objects that reference inst and mark them _p_changed. I'm in zeo land. When I do this in one client things work fine. More careful testing would show that not to be the case, I think. But of course the other clients who have already loaded the instance still hold the old class. Restarting them gets the new class so it did get stored properly in the pickle (which I would expect). The class is also stored in every reference to that object. Any chance I can get zeo to tell the other clients about the new class? Marius Gedminas -- It's my understanding that although in principle TCP can handle huge throughputs in practice many stacks haven't been optimized for that case, so you have to either use a utility which opens multiple TCP sessions in parallel or do something really radical like upgrade to the latest version of the linux kernel. -- Bram Cohen signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Writing Persistent Class
On Mon, Jan 21, 2008 at 07:15:42PM +0100, Dieter Maurer wrote: Marius Gedminas wrote at 2008-1-21 00:08 +0200: Personally, I'd be afraid to use deepcopy on a persistent object. A deepcopy is likely to be no copy at all. As Python's deepcopy does not know about object ids, it is likely that the copy result uses the same oids as the original. When you store this copy, objects with the same oid are identified. This appears not to be the case. The following script prints Looks OK: #!/usr/bin/python import copy import transaction from persistent import Persistent from ZODB.DB import DB from ZODB.MappingStorage import MappingStorage class SampleObject(Persistent): pass db = DB(MappingStorage()) conn = db.open() conn.root()['obj1'] = SampleObject() transaction.commit() # Use a different connection to sidestep the ZODB object cache conn2 = db.open() conn2.root()['obj2'] = copy.deepcopy(conn2.root()['obj1']) transaction.commit() conn3 = db.open() obj1 = conn3.root()['obj1'] obj2 = conn3.root()['obj2'] transaction.commit() if obj1 is obj2: print Fail: instead of a copy we got the same object elif obj1._p_oid == obj2._p_oid: print Fail: copy has the same oid else: print Looks OK. Marius Gedminas -- I used to think I was indecisive, but now I'm not so sure. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] PGStorage
On Tue, Jan 22, 2008 at 05:43:42PM -0200, Flavio Coelho wrote: Actually what I am trying to run away from is the packing monster ;-) I want to be able to use an OO database without the inconvenience of having it growing out of control and then having to spend hours packing the database every once in a while. (I do a lot of writes in my DBs). Do this Holy grail of databases exist? :-) I've learned to love this aspect of FileStorage. Sure, the Data.fs is measured in gigabytes, but when the users run to you crying help! help! why is this bit of data not what I expected it to be? you can dig into object history from a debugzope console and figure what changed it and when. Or restore deleted data, for that matter. Marius Gedminas -- MSDOS didn't get as bad as it is overnight -- it took over ten years of careful development. -- [EMAIL PROTECTED] signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Writing Persistent Class
On Thu, Jan 17, 2008 at 07:08:19PM -0600, Kenneth Miller wrote: There error i receive seems to have a problem with my __eq__ method. Here's the error I get when I do subclass persistent: ...No handlers could be found for logger ZODB.Connection E == ERROR: testPersistence (__main__.ToolDataTests) -- Traceback (most recent call last): File Objects.py, line 1412, in testPersistence self.failUnlessEqual(readFromFS(),td) File /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/unittest.py, line 332, in failUnlessEqual if not first == second: File Objects.py, line 1278, in __eq__ Below is where it's choking on my __eq__ method!!! return self.name == other.name and self.value == other.value and mx.DateTime.cmp(self.timeStamp,other.timeStamp,Globals.AvgTimePrec)==0 and self.slowData == other.slowData File /Library/Python/2.5/site-packages/ZODB3-3.7.2-py2.5-macosx-10.5-i386.egg/ZODB/Connection.py, line 758, in setstate raise ConnectionStateError(msg) ConnectionStateError: Shouldn't load state for 0x01 when the connection is closed It seems that you're keeping a reference to a persistent object after you close the ZODB connection. Don't do that. Marius Gedminas -- If Linux doesn't have the solution, you have the wrong problem. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Error starting simplest ZEO instance
On Sun, Aug 12, 2007 at 07:01:11PM +, Jim Carroll wrote: Hi I'm having trouble getting my web server started. It's starting ZEO, but zeo never completes the startup process. The zdctl.py is wating forever for a runzeo which fails instantly with error code 2. Im running ZODB 3.3.1 on Python 2.4 on Ubuntu. I've tried a stand-alone startup to see what's going on and it tells me: [EMAIL PROTECTED]:~/tmp# zdctl.py -p runzeo.py -a -f Data.fs start Error: invalid value for -m '18': invalid literal for int(): 18 Huh. I stepped through runzeo (and zdctl) with pdb, and figured out that it's actually running: /usr/bin/python /usr/bin/zdrun.py -S schema.xml -b 10 -s zdsock -u www -x 0,2 runzeo.py -a -f Data.fs -t 10 -m 18 I'd be inclined to look for invisible control characters such as CR at the end of that line. Marius Gedminas -- The moral of the story is that with a contrived example, you can prove anything. Oops. No, that's not what I meant to say. -- Joel Spolski signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Sun, Feb 11, 2007 at 07:29:58PM -0500, Manuel Vazquez Acosta wrote: The other question is about compiling ZODB without using the out-of-the-box distutils installation procedure. I'm also playing with Zope and Plone, so I have several instances on the same machine. I think installing with distutils may cause conflicts with the Zope instances. Am I right? If so, then how should I install ZODB side-by-side the consumer-producer application? This sounds like a job for zc.buildout. Marius Gedminas -- Special bonus feature: absolutely nowhere in RetchMail's code is there an arbitrary 3-second sleep(). Wow! What other mail retriever can say that? (Hint: not fetchmail.) -- http://open.nit.ca/wiki/index.php?page=RetchMail signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Calculating Physical Object Size
On Tue, Aug 08, 2006 at 04:19:10PM -0400, Chris S wrote: Is there anyway to determine how much disk space a specific object (including child objects) occupies in filestorage? For argument's sake, I'm assuming farily simplistic structures with no circular references. There's a small program floating around that does something like that. analyze.py by Matt Kromer. I think I got it from this list, or maybe it comes with Zope 2. Or both ;) Marius Gedminas -- ... one of the main causes of the fall of the Roman Empire was that, lacking zero, they had no way to indicate successful termination of their C programs. -- Robert Firth signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
