Re: [ZODB-Dev] Threads and Connections
On Jul 26, 2006, at 12:52 PM, Chris McDonough wrote: FWIW I believe by default at least, open ZODB connections are tied to the thread which did the opening (they are returned to a pool when closed and reused, possibly in another thread). And indeed each connection does have a cache; caching is one of the primary responsibilities of a connection object. See the documentation in ZODB/interfaces.py for IConnection. I *think* what is happening here is that you are committing the transaction devoted to the current connection/thread, and trying to close a connection that has pending changes from another thread. So when you do transaction.abort(), it's aborting the transaction involving the current thread, not the one associated with the other connection. That said, I'm not entirely sure what to tell you to do here; there used to be an API named setLocalTransaction or somesuch that allowed you to control the one-thread-per-connection policy wrt to a transaction minimally, IIRC. This API has disappeared, probably replaced with something more flexible, but I'm not sure what that is. I suspect it may have something to do with the transaction_manager parameter to DB.open() however. Open the database with an explicit transaction_manager if you want to manage these yourself, instead of using the thread default. If you have a connection, get the transaction_manager off of the connection. Can't find it in the interface, unfortunately, but I believe it to be reliable. All that said, Chris S, it sounds like you might be attacking a solved problem--and *might* be adding some serious unnecessary complexity. Maybe you ought to take a step back and see if anyone has a general approach to what you want? For instance, you said The reason why I'm doing this is because I'm trying to update the classes of persistent objects loaded into memory. Want to elaborate on this a bit? It sounds like stuff other folks have done, but you might want to give more detail. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads and Connections
Chris S wrote at 2006-7-25 23:40 -0400: The problem is the only objects I need to grab are the ones loaded by other connections. I use Python's garbage collector to track references to instances of classes that have been updated. Use a customized ZODB.Conntection.Connection class instead. Override its register method as needed. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads and Connections
On 7/26/06, Gary Poster [EMAIL PROTECTED] wrote: The reason why I'm doing this is because I'm trying to update the classes of persistent objects loaded into memory. Want to elaborate on this a bit? It sounds like stuff other folks have done, but you might want to give more detail. When a file containing source code is modified, I want to reload it using Python's reload() function, then update classes with the new definitions. I have application-level logic to ensure the objects are in a sane-state during update. The only problem I'm having is when I close the connection after an update, I get ConnectionStateError, Cannot close a connection joined to a transaction. Setting conn._needs_to_join = True appears to fix this, but it's too much of a hack and I have to test it further. There was similar discussion regarding the reloading product over a year ago, but this functionality is specific to Zope, which I'm not using. Also, it seems unmaintained and unreliable, and my have been deprecated. Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads and Connections
On 7/26/06, Chris McDonough [EMAIL PROTECTED] wrote: FWIW I believe by default at least, open ZODB connections are tied to the thread which did the opening (they are returned to a pool when closed and reused, possibly in another thread). And indeed each connection does have a cache; caching is one of the primary responsibilities of a connection object. See the documentation in ZODB/interfaces.py for IConnection. I *think* what is happening here is that you are committing the transaction devoted to the current connection/thread, and trying to close a connection that has pending changes from another thread. So when you do transaction.abort(), it's aborting the transaction involving the current thread, not the one associated with the other connection. That said, I'm not entirely sure what to tell you to do here; there used to be an API named setLocalTransaction or somesuch that allowed you to control the one-thread-per-connection policy wrt to a transaction minimally, IIRC. This API has disappeared, probably replaced with something more flexible, but I'm not sure what that is. I suspect it may have something to do with the transaction_manager parameter to DB.open() however. Thanks, using a single transaction manager along with setting conn._needs_to_join = True right before I close the connection appears to fix the problem. It's a bit of a hack, but it works. Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Invalid object references on brand new Data.fs using fsrefs.py
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Ross, Troy D wrote:
Hello,
I just created a brand new instance of a Zope 2.9.3, started the
instance once so that it creates a Data.fs file and then gracefully shut
down the zope server. I then ran fsrefs.py on Data.fs and I was given
the following results:
oid 0x0L persistent.mapping.PersistentMapping
last updated: 2006-07-26 21:48:47.904932, tid=0x36714FCCC64F5DDL
refers to invalid objects:
oid ('\x00\x00\x00\x00\x00\x00\x00\x01', None) missing:
'unknown'
oid ('\x00\x00\x00\x00\x00\x00\x00\n', None) missing:
'unknown'
...
oid 0xA79L BTrees._IOBTree.IOBTree
last updated: 2006-07-26 21:48:54.163685, tid=0x36714FCE7193011L
refers to invalid objects:
oid ('\x00\x00\x00\x00\x00\x00\nz', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\x00\n{', None) missing: 'unknown'
oid ('\x00\x00\x00\x00\x00\x00\n|', None) missing: 'unknown'
Similar load errors were repeated for a total output of 6713 lines.
Are these references that don't refer to pickled objects supposed to be
in the Data.fs file? I've run earlier version of zope and I never saw
such load errors. The system is running FreeBSD 6.1 with python 2.4.3.
I tested a similar setup on a Windows XP machine with similar results.
I can reproduce this on the Zope 2.9 branch, but not on the 2.8 branch.
$ export ZHOME=/path/to/zope
$ $ZHOME/bin/mkzopeinstance.py -d /tmp/initrefs -u admin:123
$ cd /tmp/initrefs
$ bin/zopectl fg
...
2006-07-24 20:34:27 INFO Zope Ready to handle requests
^C
$ PYTHONPATH=$ZHOME/lib/python \
$ZHOME/bin/fsrefs var/Data.fs
My *guess* is that this represents a failure in the fsrefs script, since
the appserver is functioning properly. Perhaps it has not been updated
to deal with the API changes between ZODB 3.4.x and ZODB 3.6.x? The
delta between the two is pretty small::
$ svn diff $ZSVN/ZODB/branches/3.{4,6}/src/scripts/fsrefs.py
- --- fsrefs.py (.../3.4/src/scripts/fsrefs.py) (revision 69267)
+++ fsrefs.py (.../3.6/src/scripts/fsrefs.py) (revision 69267)
@@ -68,7 +68,8 @@
from ZODB.FileStorage import FileStorage
from ZODB.TimeStamp import TimeStamp
- -from ZODB.utils import u64, oid_repr, get_refs, get_pickle_metadata
+from ZODB.utils import u64, oid_repr, get_pickle_metadata
+from ZODB.serialize import get_refs
from ZODB.POSException import POSKeyError
VERBOSE = 0
@@ -129,9 +130,8 @@
refs = get_refs(data)
missing = [] # contains 3-tuples of oid, klass-metadata, reason
for info in refs:
- -try:
- -ref, klass = info
- -except (ValueError, TypeError):
+ref, klass = info
+if klass is None:
# failed to unpack
ref = info
klass = 'unknown'
Which does play in the output we see on the 3.6 branch.
Tres.
- --
===
Tres Seaver +1 202-558-7113 [EMAIL PROTECTED]
Palladion Software Excellence by Designhttp://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFEyAzv+gerLs4ltQ4RAtatAKDa2x4rnMpVGxrMHWpb4mEOFcRrCQCdFMBB
XO39Y+Xk11LVLAjjI1/I38M=
=dYiT
-END PGP SIGNATURE-
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Invalid object references on brand new Data.fs using fsrefs.py
[various fsrefs.py failures in Zope 2.9.3]
[Tres]
I can reproduce this on the Zope 2.9 branch, but not on the 2.8 branch.
I'll note that it can be reproduced easily with a standalone ZODB 3.6 too:
import ZODB
import ZODB.FileStorage
import transaction
from BTrees.OOBTree import OOBTree
st = ZODB.FileStorage.FileStorage(Data.fs)
db = ZODB.DB(st)
cn = db.open()
rt = cn.root()
rt[tree] = OOBTree()
transaction.commit()
db.close()
Run that, and then:
$ python24\python.exe src\scripts\fsrefs.py Data.fs
oid 0x0L persistent.mapping.PersistentMapping
last updated: 2006-07-27 01:37:35.342000, tid=0x36715E196CAE377L
refers to invalid object:
oid ('\x00\x00\x00\x00\x00\x00\x00\x01', None) missing: 'unknown'
...
My *guess* is that this represents a failure in the fsrefs script, since
the appserver is functioning properly. Perhaps it has not been updated
to deal with the API changes between ZODB 3.4.x and ZODB 3.6.x? The
delta between the two is pretty small::
I expect it was actually the /entire/ checkin that made that change to
fsrefs.py:
http://svn.zope.org/ZODB/trunk/src/ZODB/serialize.py?rev=30715view=rev
That moved and changed the crucial get_refs() function used by
fsrefs.py too. Unfortunately, I don't believe there are any tests for
fsrefs, and I bet it's been broken ever since that checkin.
This code in fsrefs.py doesn't make sense now:
ref, klass = info
if klass is None:
# failed to unpack
ref = info
klass = 'unknown'
The failed to unpack comment is left over from before that checkin,
where it did make sense. The ref = info after it appears to be dead
wrong. Try this patch?:
Index: src/scripts/fsrefs.py
===
--- src/scripts/fsrefs.py (revision 69267)
+++ src/scripts/fsrefs.py (working copy)
@@ -129,11 +129,8 @@
data, serial = fs.load(oid, )
refs = get_refs(data)
missing = [] # contains 3-tuples of oid, klass-metadata, reason
-for info in refs:
-ref, klass = info
+for ref, klass in refs:
if klass is None:
-# failed to unpack
-ref = info
klass = 'unknown'
if ref not in fs._index:
missing.append((ref, klass, missing))
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
