Re: [ZODB-Dev] ZODB file never updated - import transaction fails...
[EMAIL PROTECTED] wrote at 2006-8-2 15:36 -0500: Dieter I could only imagine that the transaction you commit is not the Dieter transaction the modifications are registered with. Dieter This could, e.g., happen when you modify the objects in a thread Dieter different from where the transaction is committed (the default Dieter transaction manager manages a transaction per thread). However, Dieter you code does not give a hint that this is indeed the problem. That's not the case here. Some SpamBayes apps use threads, but my train-to-exhaustion script does not. Then, you might want to debug what happens. The basic operation is as follows: When you modify a persistent object that was not yet registered with the connection, the connection's register method is called. register registers the connection with the (current) transaction as a ResourceManager (if not already done) and then registers the object with the connection. An object already registered with the connection has obj._p_changed == 1. When the transaction commits, it commits all its registered ResourceManagers which in turn commit all their registered objects. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB file never updated - import transaction fails...
[EMAIL PROTECTED] wrote at 2006-7-30 11:06 -0500:
...
try:
import dnscache
cache = dnscache.cache(cachefile=os.path.expanduser(~/.dnscache))
cache.printStatsAtEnd = True
except (IOError, ImportError):
cache = None
else:
import atexit
atexit.register(cache.close)
print calling, cache.close, at exit
It dutifully prints calling bound method cache.close of
spambayes.dnscache.cache instance at 0x119a4b8 at exit at startup. I
added a print to the close method. It prints closing zodb at the right
time. There is no longer any complaint about importing the transaction
module. As far as I can tell, it should be working. However, when I look
at the timestamps on the various files, it appears only the .dnscache.index
file has been updated:
% ls -ltr ~/.dnscache*
-rw-rw-r-- 1 skip staff 44547 Jul 23 09:06 /Users/skip/.dnscache.old
-rw-rw-r-- 1 skip staff 0 Jul 23 09:07 /Users/skip/.dnscache.tmp
-rw-rw-r-- 1 skip staff 44383 Jul 29 08:22 /Users/skip/.dnscache
-rw-rw-r-- 1 skip staff293 Jul 29 18:13 /Users/skip/.dnscache.index
(It's about 18:20 here right now.)
So this code appears to be getting called and executes to completion:
def _zodb_store(self):
import transaction
from ZODB.POSException import ConflictError
from ZODB.POSException import TransactionFailedError
try:
transaction.commit()
except ConflictError, msg:
# We'll save it next time, or on close. It'll be lost if we
# hard-crash, but that's unlikely, and not a particularly big
# deal.
if options[globals, verbose]:
print sys.stderr, Conflict on commit., msg
transaction.abort()
except TransactionFailedError, msg:
# Saving isn't working. Try to abort, but chances are that
# restarting is needed.
if options[globals, verbose]:
print sys.stderr, Store failed. Need to restart., msg
transaction.abort()
def _zodb_close(self):
# Ensure that the db is saved before closing. Alternatively, we
# could abort any waiting transaction. We need to do *something*
# with it, though, or it will be still around after the db is
# closed and cause problems. For now, saving seems to make sense
# (and we can always add abort methods if they are ever needed).
self._zodb_store()
# Do the closing.
self._DB.close()
# We don't make any use of the 'undo' capabilities of the
# FileStorage at the moment, so might as well pack the database
# each time it is closed, to save as much disk space as possible.
# Pack it up to where it was 'yesterday'.
# XXX What is the 'referencesf' parameter for pack()? It doesn't
# XXX seem to do anything according to the source.
self._zodb_storage.pack(time.time()-60*60*24, None)
self._zodb_storage.close()
self._zodb_closed = True
if options[globals, verbose]:
print sys.stderr, 'Closed dnscache database'
There are no prints about conflicts or failed stores. No import errors, the
final Closed dnscache database line is output. Still, there is no update
of the actual data on disk. Recalling this code:
from ZODB import DB
from ZODB.FileStorage import FileStorage
self._zodb_storage = FileStorage(cachefile, read_only=False)
self._DB = DB(self._zodb_storage, cache_size=1)
self._conn = self._DB.open()
root = self._conn.root()
self.caches = root.get(dnscache)
if self.caches is None:
# There is no classifier, so create one.
from BTrees.OOBTree import OOBTree
self.caches = root[dnscache] = OOBTree()
self.caches[A] = {}
self.caches[PTR] = {}
do I need to do something with self.caches or with the root object to
actually flush the data to the file?
In principle, this is not necessary.
What you see is very strange.
I could only imagine that the transaction you commit is not the
transaction the modifications are registered with.
This could, e.g., happen when you modify the objects in a thread
different from where the transaction is committed (the default
transaction manager manages a transaction per thread).
However, you code does not give a hint that this is indeed
the problem.
--
Viele Grüße
Dieter
Tel: 0681-761 8751 bzw. 06894-870 177
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB file never updated - import transaction fails...
Dieter What you see is very strange. That's strangely heartening. ;-) Dieter I could only imagine that the transaction you commit is not the Dieter transaction the modifications are registered with. Dieter This could, e.g., happen when you modify the objects in a thread Dieter different from where the transaction is committed (the default Dieter transaction manager manages a transaction per thread). However, Dieter you code does not give a hint that this is indeed the problem. That's not the case here. Some SpamBayes apps use threads, but my train-to-exhaustion script does not. Skip ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB file never updated - import transaction fails...
Chris The first thing I'd try to do is to move the cleanup code out of
Chris __del__ (and perhaps to a finally: after the line that creates
Chris the cache instead) and see if that helps.
Dieter Avoid using __del__ and call your close explicitely.
Good point about __del__. I changed the import of the dnscache code to
this:
try:
import dnscache
cache = dnscache.cache(cachefile=os.path.expanduser(~/.dnscache))
cache.printStatsAtEnd = True
except (IOError, ImportError):
cache = None
else:
import atexit
atexit.register(cache.close)
print calling, cache.close, at exit
It dutifully prints calling bound method cache.close of
spambayes.dnscache.cache instance at 0x119a4b8 at exit at startup. I
added a print to the close method. It prints closing zodb at the right
time. There is no longer any complaint about importing the transaction
module. As far as I can tell, it should be working. However, when I look
at the timestamps on the various files, it appears only the .dnscache.index
file has been updated:
% ls -ltr ~/.dnscache*
-rw-rw-r-- 1 skip staff 44547 Jul 23 09:06 /Users/skip/.dnscache.old
-rw-rw-r-- 1 skip staff 0 Jul 23 09:07 /Users/skip/.dnscache.tmp
-rw-rw-r-- 1 skip staff 44383 Jul 29 08:22 /Users/skip/.dnscache
-rw-rw-r-- 1 skip staff293 Jul 29 18:13 /Users/skip/.dnscache.index
(It's about 18:20 here right now.)
So this code appears to be getting called and executes to completion:
def _zodb_store(self):
import transaction
from ZODB.POSException import ConflictError
from ZODB.POSException import TransactionFailedError
try:
transaction.commit()
except ConflictError, msg:
# We'll save it next time, or on close. It'll be lost if we
# hard-crash, but that's unlikely, and not a particularly big
# deal.
if options[globals, verbose]:
print sys.stderr, Conflict on commit., msg
transaction.abort()
except TransactionFailedError, msg:
# Saving isn't working. Try to abort, but chances are that
# restarting is needed.
if options[globals, verbose]:
print sys.stderr, Store failed. Need to restart., msg
transaction.abort()
def _zodb_close(self):
# Ensure that the db is saved before closing. Alternatively, we
# could abort any waiting transaction. We need to do *something*
# with it, though, or it will be still around after the db is
# closed and cause problems. For now, saving seems to make sense
# (and we can always add abort methods if they are ever needed).
self._zodb_store()
# Do the closing.
self._DB.close()
# We don't make any use of the 'undo' capabilities of the
# FileStorage at the moment, so might as well pack the database
# each time it is closed, to save as much disk space as possible.
# Pack it up to where it was 'yesterday'.
# XXX What is the 'referencesf' parameter for pack()? It doesn't
# XXX seem to do anything according to the source.
self._zodb_storage.pack(time.time()-60*60*24, None)
self._zodb_storage.close()
self._zodb_closed = True
if options[globals, verbose]:
print sys.stderr, 'Closed dnscache database'
There are no prints about conflicts or failed stores. No import errors, the
final Closed dnscache database line is output. Still, there is no update
of the actual data on disk. Recalling this code:
from ZODB import DB
from ZODB.FileStorage import FileStorage
self._zodb_storage = FileStorage(cachefile, read_only=False)
self._DB = DB(self._zodb_storage, cache_size=1)
self._conn = self._DB.open()
root = self._conn.root()
self.caches = root.get(dnscache)
if self.caches is None:
# There is no classifier, so create one.
from BTrees.OOBTree import OOBTree
self.caches = root[dnscache] = OOBTree()
self.caches[A] = {}
self.caches[PTR] = {}
do I need to do something with self.caches or with the root object to
actually flush the data to the file?
One other thing I discovered this morning. If I delete the ~/.dnscache* and
rerun my SpamBayes training, it obligingly creates a new set of files. If I
run it again though it doesn't update ~/.dnscache, only ~/.dnscache.tmp and
~/.dnscache.index.
Thx,
--
Skip Montanaro - [EMAIL PROTECTED] - http://www.mojam.com/
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB file never updated - import transaction fails...
Since your exception is happening during __del__ and since AFAIK no
guarantees are made as to what will exist is sys.modules during any
given object's __del__, it's maybe not suprising that you're getting
this exception. I don't know that this has anything to do with ZODB
in particular.
The first thing I'd try to do is to move the cleanup code out of
__del__ (and perhaps to a finally: after the line that creates the
cache instead) and see if that helps.
- C
On Jul 29, 2006, at 9:40 AM, [EMAIL PROTECTED] wrote:
Matt Cowles has a DNS cache module with a patch for use with
SpamBayes. It
doesn't persist the data (just caches during the lifetime of the
run). I'm
trying to add persistence to that cache and wanting to use the same
database
as the user selects for the main SpamBayes database, my first two
targets
were anydbm and ZODB. I'm having trouble with the ZODB setup. It
seemed to
work well at the start, but the database file is never updated when
I rerun
my training script, so its not caching anything, and now everything
in the
database has expired. I've never used ZODB before. The amount of
code is
fairly small, so I'll just post it here in hopes someone can see
what I've
done wrong.
When opening the database it selects based on the user-preferred
filetype.
For zodb it executes:
from ZODB import DB
from ZODB.FileStorage import FileStorage
self._zodb_storage = FileStorage(cachefile, read_only=False)
self._DB = DB(self._zodb_storage, cache_size=1)
self._conn = self._DB.open()
root = self._conn.root()
self.caches = root.get(dnscache)
if self.caches is None:
# There is no classifier, so create one.
from BTrees.OOBTree import OOBTree
self.caches = root[dnscache] = OOBTree()
self.caches[A] = {}
self.caches[PTR] = {}
That gives the cache a caches attribute that looks like the
dictionary it
uses in the no persistence case.
When the cache is deleted, its __del__ method calls
self.close()
which calls self._zodb_close():
def _zodb_close(self):
# Ensure that the db is saved before closing. Alternatively, we
# could abort any waiting transaction. We need to do
*something*
# with it, though, or it will be still around after the db is
# closed and cause problems. For now, saving seems to make
sense
# (and we can always add abort methods if they are ever needed).
self._zodb_store()
# Do the closing.
self._DB.close()
which calls self._zodb_store():
def _zodb_store(self):
import transaction
from ZODB.POSException import ConflictError
from ZODB.POSException import TransactionFailedError
try:
transaction.commit()
except ConflictError, msg:
# We'll save it next time, or on close. It'll be lost if we
# hard-crash, but that's unlikely, and not a particularly
big
# deal.
if options[globals, verbose]:
print sys.stderr, Conflict on commit., msg
transaction.abort()
except TransactionFailedError, msg:
# Saving isn't working. Try to abort, but chances are that
# restarting is needed.
if options[globals, verbose]:
print sys.stderr, Store failed. Need to restart.,
msg
transaction.abort()
which seems to fail. I get this message in the log:
Exception exceptions.ImportError: 'No module named transaction' in
bound method cache.__del__ of spambayes.dnscache.cache
instance at
0x11994b8 ignored
I can import transaction from the interpreter prompt just fine:
% python
Python 2.5b2 (trunk:50921, Jul 28 2006, 20:21:50)
[GCC 4.0.0 (Apple Computer, Inc. build 5026)] on darwin
Type help, copyright, credits or license for more
information.
import transaction
transaction.__file__
'/Users/skip/local/lib/python2.5/site-packages/transaction/
__init__.pyc'
import ZODB
ZODB.__file__
'/Users/skip/local/lib/python2.5/site-packages/ZODB/__init__.pyc'
I stuck a
print transaction.__file__
after the import in _zodb_close and ran dnscache as a main. It
prints out
/Users/skip/local/lib/python2.5/site-packages/transaction/
__init__.pyc
so I'm pretty sure I'm getting the same version of Python. Any
idea why it
would fail in one instance but not another? The only thing I can
think of
is that there are two ZODB databases opened at the same time in the
real use
case, but only the one in the situation where I'm running the
dnscache test
function. Is there something about having two distinct ZODB
database files
open I need to consider?
Thanks,
--
Skip Montanaro - [EMAIL PROTECTED] - http://www.mojam.com/
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
Re: [ZODB-Dev] ZODB file never updated - import transaction fails...
[EMAIL PROTECTED] wrote at 2006-7-29 08:40 -0500: ... When the cache is deleted, its __del__ method calls self.close() which calls self._zodb_close(): def _zodb_close(self): # Ensure that the db is saved before closing. Alternatively, we # could abort any waiting transaction. We need to do *something* # with it, though, or it will be still around after the db is # closed and cause problems. For now, saving seems to make sense # (and we can always add abort methods if they are ever needed). self._zodb_store() # Do the closing. self._DB.close() which calls self._zodb_store(): def _zodb_store(self): import transaction ... which seems to fail. I get this message in the log: Exception exceptions.ImportError: 'No module named transaction' in bound method cache.__del__ of spambayes.dnscache.cache instance at 0x11994b8 ignored Is the cache deleted when Python terminates? At this time, Python no longer works reliable as resources must be released in some order and after a resource is released it cannot be used anymore. In your case, sys.path might have been released when the destructor tries to import transaction. It will fail then. Avoid using __del__ and call your close explicitely. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
