Re: [ZODB-Dev] Progress report: porting persistent, BTrees to Python3
On Sun, 2012-12-16 at 22:10 +0100, Godefroid Chapelle wrote: Le 15/12/12 01:52, Tres Seaver a écrit : I fixed the remainig issues in persistent and released 4.0.5 today: its tests properly exercise the C extensions Under Python 3.2 / 3.3. I want to express my thanks to you, Tres, for taking care of that work ! This port of ZODB to Python 3 is really a crucial step for the ZTK ecosystem. After the work already done on zope.interface and zope.component. Further, I'd like to also thank Jim for his work on porting buildout. When this will be finished, porting the rest of the ZTK should be much easier, which hopefully implies that more of us will be able to participate. Hear, hear! - C ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction Attempts class
On Sat, 2012-03-31 at 07:49 -0400, Jim Fulton wrote:
On Thu, Mar 29, 2012 at 6:59 PM, Chris McDonough chr...@plope.com wrote:
On Thu, 2012-03-29 at 07:18 -0400, Jim Fulton wrote:
On Wed, Mar 28, 2012 at 6:37 PM, Chris McDonough chr...@plope.com wrote:
On Wed, 2012-03-28 at 17:06 -0400, Jim Fulton wrote:
On Wed, Mar 28, 2012 at 4:37 PM, Chris McDonough chr...@plope.com
wrote:
On Wed, 2012-03-28 at 14:21 -0400, Jim Fulton wrote:
...
A decorator for running some code in the context of a txn and retrying
retryable exceptions would be nice higher level behavior. Â I'd be
willing to do this work over this weekend.
Cool. Don't forget the transaction note part. :)
Too many transactions in our apps don't have
notes, especially transactions that happen
outside of web requests.
In the meantime, I think the existing attempts context manager still
needs the small fix I proposed in my original message. Â Can you
confirm
that my understanding of the its intent seems roughly correct?
I didn't see a statement of intent. I think your fix us good,
but the code (that I wrote and your fix) makes my head hurt. :)
I'd say, make some test cases for the bug and make it pass.
OK. Once I fix this Attempts bug, I think the decorator code is just
a higher level interface that uses it:
class job(object):
def __init__(self, attempts=1, note=None, manager=None):
self.attempts = attempts
self.note = note
if manager is None:
manager = transaction.manager
self.manager = manager
def __call__(self, wrapped):
note = self.note
if note is None:
note = getattr(wrapped, '__name__', None)
def inner(*arg, **kw):
for attempt in self.manager.attempts(self.attempts):
with attempt as t:
t.note(note)
return wrapped(*arg, **kw)
.. or something like that...
It could be written that way, although I would
use a much simpler implementation.
The attempt design was a reach to overcome
the limitations of the with statement. I'm not at all happy
with it, although I couldn't think of anything better at the
time. I hate to build on it.
It's brainbusting, yes, but it works. Do you have any other specific
pattern in mind?
def job(func=None, retries=3):
if func is None:
return lambda f: job(f, retries)
note = func.__doc__
if note:
note = note.split('\n', 1)[0]
else:
note = func.__name__
for i in xrange(retries + 1):
t = manager.begin()
if i:
t.note(%s (retry: %s) % (note, i))
else:
t.note(note)
try:
func(t)
t.commit()
except TransientError:
t.abort()
else:
break
The above is untested, but you get the idea.
Aside from note and decorator support,
this is straightforward. It encapsulates
standard boilerplate.
Alright, I took a stab at it with tests, albeit as a method of the
TransactionManager class (and an alias in __init__ as job):
https://mail.zope.org/pipermail/checkins/2012-April/059120.html
- C
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction Attempts class
On Wed, 2012-03-28 at 18:37 -0400, Chris McDonough wrote: On Wed, 2012-03-28 at 17:06 -0400, Jim Fulton wrote: On Wed, Mar 28, 2012 at 4:37 PM, Chris McDonough chr...@plope.com wrote: On Wed, 2012-03-28 at 14:21 -0400, Jim Fulton wrote: ... A decorator for running some code in the context of a txn and retrying retryable exceptions would be nice higher level behavior. Â I'd be willing to do this work over this weekend. Cool. Don't forget the transaction note part. :) Too many transactions in our apps don't have notes, especially transactions that happen outside of web requests. In the meantime, I think the existing attempts context manager still needs the small fix I proposed in my original message. Â Can you confirm that my understanding of the its intent seems roughly correct? I didn't see a statement of intent. I think your fix us good, but the code (that I wrote and your fix) makes my head hurt. :) I'd say, make some test cases for the bug and make it pass. https://mail.zope.org/pipermail/checkins/2012-March/059068.html Look reasonable? If so I'll go ahead and merge it and base the decorator code on it. - C ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction Attempts class
On Thu, 2012-03-29 at 07:18 -0400, Jim Fulton wrote: On Wed, Mar 28, 2012 at 6:37 PM, Chris McDonough chr...@plope.com wrote: On Wed, 2012-03-28 at 17:06 -0400, Jim Fulton wrote: On Wed, Mar 28, 2012 at 4:37 PM, Chris McDonough chr...@plope.com wrote: On Wed, 2012-03-28 at 14:21 -0400, Jim Fulton wrote: ... A decorator for running some code in the context of a txn and retrying retryable exceptions would be nice higher level behavior. Â I'd be willing to do this work over this weekend. Cool. Don't forget the transaction note part. :) Too many transactions in our apps don't have notes, especially transactions that happen outside of web requests. In the meantime, I think the existing attempts context manager still needs the small fix I proposed in my original message. Â Can you confirm that my understanding of the its intent seems roughly correct? I didn't see a statement of intent. I think your fix us good, but the code (that I wrote and your fix) makes my head hurt. :) I'd say, make some test cases for the bug and make it pass. OK. Once I fix this Attempts bug, I think the decorator code is just a higher level interface that uses it: class job(object): def __init__(self, attempts=1, note=None, manager=None): self.attempts = attempts self.note = note if manager is None: manager = transaction.manager self.manager = manager def __call__(self, wrapped): note = self.note if note is None: note = getattr(wrapped, '__name__', None) def inner(*arg, **kw): for attempt in self.manager.attempts(self.attempts): with attempt as t: t.note(note) return wrapped(*arg, **kw) .. or something like that... It could be written that way, although I would use a much simpler implementation. The attempt design was a reach to overcome the limitations of the with statement. I'm not at all happy with it, although I couldn't think of anything better at the time. I hate to build on it. It's brainbusting, yes, but it works. Do you have any other specific pattern in mind? - C ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] transaction Attempts class
The transaction package offers a nice feature, where you can say: import transaction for attempt in transaction.attempts(3): with attempt as t: ... do something ... If do something raises a ConflictError (or any other retryable error), the next attempt is tried, until all attempts have been exhausted, at which point, it gives up and the exception is raised. But I think it may be slightly broken. Here's the definition of the attempt context manager: class Attempt(object): def __init__(self, manager): self.manager = manager def __enter__(self): return self.manager.__enter__() def __exit__(self, t, v, tb): if v is None: self.manager.commit() else: retry = self.manager._retryable(t, v) self.manager.abort() return retry do_something within the body of an attempt context manager usually doesn't raise a retryable exception (it's business logic), but the self.manager.commit() within the __exit__ of the context manager usually does. When this happens, nothing actually catches the exception. I think the context manager may need to be changed to this: class Attempt(object): def __init__(self, manager): self.manager = manager def __enter__(self): return self.manager.__enter__() def __exit__(self, t, v, tb): if v is None: try: self.manager.commit() except: retry = self.manager._retryable(*sys.exc_info()[:2]) self.manager.abort() return retry else: retry = self.manager._retryable(t, v) self.manager.abort() return retry Either that or it needs to not try to do a commit itself, and leave it up to the caller. Anybody with thoughts? - C ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction Attempts class
On Wed, 2012-03-28 at 08:33 +0200, Thierry Florac wrote: I used this feature just a few days ago. It was in a batch application where conflict errors can frequently occur, and they **seemed** to be handled correctly. I doubt it's working correctly if you have any retryable errors happening during commit. My only problem is that I had to add a break at the end of my for loop. Otherwise the transaction was committed several times and the mail report which was sent at the end of the batch was also sent several times... Did I missed something ? Otherwise perhaps the 'transaction' package documentation should be updated ?? Not sure what you'd say there.. a for loop has to be broken out of for the for loop to not loop? ;-) - C ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction Attempts class
On Wed, 2012-03-28 at 17:06 -0400, Jim Fulton wrote: On Wed, Mar 28, 2012 at 4:37 PM, Chris McDonough chr...@plope.com wrote: On Wed, 2012-03-28 at 14:21 -0400, Jim Fulton wrote: ... A decorator for running some code in the context of a txn and retrying retryable exceptions would be nice higher level behavior. Â I'd be willing to do this work over this weekend. Cool. Don't forget the transaction note part. :) Too many transactions in our apps don't have notes, especially transactions that happen outside of web requests. In the meantime, I think the existing attempts context manager still needs the small fix I proposed in my original message. Â Can you confirm that my understanding of the its intent seems roughly correct? I didn't see a statement of intent. I think your fix us good, but the code (that I wrote and your fix) makes my head hurt. :) I'd say, make some test cases for the bug and make it pass. OK. Once I fix this Attempts bug, I think the decorator code is just a higher level interface that uses it: class job(object): def __init__(self, attempts=1, note=None, manager=None): self.attempts = attempts self.note = note if manager is None: manager = transaction.manager self.manager = manager def __call__(self, wrapped): note = self.note if note is None: note = getattr(wrapped, '__name__', None) def inner(*arg, **kw): for attempt in self.manager.attempts(self.attempts): with attempt as t: t.note(note) return wrapped(*arg, **kw) .. or something like that... I'll admit, I didn't understand Either that or it needs to not try to do a commit itself, and leave it up to the caller. The caller in this case is the body of the context manager which calls __exit__ by raising an exception or returning. I just meant instead of: for attempt in transaction.attempts(3): with attempt as t: .. do stuff, rely on context manager __exit__ to commit .. You might change its contract to: for attempt in transaction.attempts(3): with attempt as t: .. do stuff .. t.commit() # you must commit yourself, __exit_ catches # errors only and aborts on exception But this would break existing code as well as probably going against the spirit of intent. - C ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] transaction release
If no one complains, I'm going to make another transaction package release this evening. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction release
Tagged and released to PyPI as 1.1.1. - C On Thu, 2010-09-16 at 15:05 -0400, Chris McDonough wrote: If no one complains, I'm going to make another transaction package release this evening. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] transaction exception-related reference leaks
I noticed some weird behavior with exceptions and transactions recently (in particular, a WSGI server which relies on a __del__ to close its client connection wasn't disconnecting from the client immediately when a conflict error was raised by ZODB, but would indeed disconnect if a conflict error had not been raised). I tracked this down to the zope transaction module leaking frame stacks via retrieving traceback objects from sys.exc_info() without subsequently removing them from local scope. With the attached patch, the issue was solved. If no one complains, I'll apply it to the transaction trunk. - C Index: transaction/_transaction.py === --- transaction/_transaction.py (revision 116336) +++ transaction/_transaction.py (working copy) @@ -329,9 +329,15 @@ self._commitResources() self.status = Status.COMMITTED except: -t, v, tb = self._saveAndGetCommitishError() -self._callAfterCommitHooks(status=False) -raise t, v, tb +t = None +v = None +tb = None +try: +t, v, tb = self._saveAndGetCommitishError() +self._callAfterCommitHooks(status=False) +raise t, v, tb +finally: +del t, v, tb else: if self._manager: self._manager.free(self) @@ -343,14 +349,18 @@ self.status = Status.COMMITFAILED # Save the traceback for TransactionFailedError. ft = self._failure_traceback = StringIO() -t, v, tb = sys.exc_info() -# Record how we got into commit(). -traceback.print_stack(sys._getframe(1), None, ft) -# Append the stack entries from here down to the exception. -traceback.print_tb(tb, None, ft) -# Append the exception type and value. -ft.writelines(traceback.format_exception_only(t, v)) -return t, v, tb +try: +t, v, tb = sys.exc_info() +# Record how we got into commit(). +traceback.print_stack(sys._getframe(1), None, ft) +# Append the stack entries from here down to the exception. +traceback.print_tb(tb, None, ft) +# Append the exception type and value. +ft.writelines(traceback.format_exception_only(t, v)) +return t, v, tb +finally: +del t, v, tb + def _saveAndRaiseCommitishError(self): t, v, tb = self._saveAndGetCommitishError() @@ -442,10 +452,13 @@ # to revert the changes in each of the resource managers. t, v, tb = sys.exc_info() try: -self._cleanup(L) +try: +self._cleanup(L) +finally: +self._synchronizers.map(lambda s: s.afterCompletion(self)) +raise t, v, tb finally: -self._synchronizers.map(lambda s: s.afterCompletion(self)) -raise t, v, tb +del t, v, tb def _cleanup(self, L): # Called when an exception occurs during tpc_vote or tpc_finish. @@ -469,26 +482,33 @@ self._synchronizers.map(lambda s: s.beforeCompletion(self)) -tb = None -for rm in self._resources: -try: -rm.abort(self) -except: -if tb is None: -t, v, tb = sys.exc_info() -self.log.error(Failed to abort resource manager: %s, - rm, exc_info=sys.exc_info()) +try: -if self._manager: -self._manager.free(self) +t = None +v = None +tb = None + +for rm in self._resources: +try: +rm.abort(self) +except: +if tb is None: +t, v, tb = sys.exc_info() +self.log.error(Failed to abort resource manager: %s, + rm, exc_info=sys.exc_info()) -self._synchronizers.map(lambda s: s.afterCompletion(self)) +if self._manager: +self._manager.free(self) -self.log.debug(abort) +self._synchronizers.map(lambda s: s.afterCompletion(self)) -if tb is not None: -raise t, v, tb +self.log.debug(abort) +if tb is not None: +raise t, v, tb +finally: +del t, v, tb + def note(self, text): text = text.strip() if self.description: @@ -542,20 +562,26 @@ self.manager.tpc_vote(txn) def abort(self, txn): +t = None +v = None tb = None -for o in self.objects: -try: -self.manager.abort(o, txn) -except: -# Capture
Re: [ZODB-Dev] The database root: It's a trap!
On Wed, 2010-05-12 at 20:16 +0100, Chris Withers wrote: Jim Fulton wrote: C. I'd really like to be able to configure what the root object is as database creation time. That way it can be application specific. That sounds reasonable, but doesn't address the default behavior. Default would be an OOBTree for me... Scales, no thinking or scary warnings required :-) I'm not sure that any particular default root object is going to solve this problem, because they're just going to put a dictionary into one of its keys and hurt themselves that way. I think the idea of a warning when a record exceeds a given size would be more generally helpful, although that of course doesn't prohibit using a different root too. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Checking the length of OOBTree
I'm not sure about the internal len variable, but I usually maintain a separate length in something that uses a BTree by using a BTrees.Length.Length object: http://svn.repoze.org/repoze.folder/trunk/repoze/folder/__init__.py On 4/8/10 11:36 AM, Leszek Syroka wrote: Hi, what is the fastest way of checking the number of elements in OOBtree. Execution time of len( OOBtree.keys() ) and len(OOBtree) is exactly the same. For big data sets execution time is unacceptable. I found out that in the implementation of OOBtree (written in C) there is a variable called 'len', which seems to contain the length of the tree. Is it possible to access that variable from the python code without modifying the source? Best regards Leszek ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Chris McDonough Agendaless Consulting, Fredericksburg VA The repoze.bfg Web Application Framework Book: http://bfg.repoze.org/book ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] undo (and storage interface) brokenness
Jim Fulton wrote: So, I repeat: is anyone actually using undo these days? I haven't used undo (not the APIs nor via any Zope UI) in years. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Cannot import name utils + class.__dict__ not accessible
I'm not sure of the exact cause of this error, but I'll note that one your
deployment looks like it's using the slow (non-C) versions of the stuff in
zope.interface, which usually means that somehow the zope.interface C
extensions
couldn't be compiled. You might try checking this from a debug prompt:
It should look like this:
from zope.interface import implementedBy
implementedBy
built-in function implementedBy
You're not using the C extensions if it looks something like this instead:
from zope.interface import implementedBy
implementedBy
function implementedBy at 0x51cf70
I'd start by getting that fixed if possible.
- C
On 6/15/09 9:33 AM, Jose Benito Gonzalez Lopez wrote:
Dear ZODB developers,
We are experiencing some errors and we'd appreciate any idea or advise.
Our web application (http://indicodemo.cern.ch) uses ZODB:
- ZODB 3.4 in production
- and ZODB 3.8.0 for testing.
Both versions work smoothly.
Now, we are moving to ZODB3 3.8.1 (in a fresh installation of the web
appl.). The installation was fine
but while navigating through the website, we get the following errors
randomly:
error1
...
[..] File /usr/lib64/python2.4/site-packages/ZEO/ClientStorage.py,
line 34, in ?\nfrom ZEO.cache import ClientCache
[..] File /usr/lib64/python2.4/site-packages/ZEO/cache.py, line 36,
in ?\nimport ZODB.fsIndex
[..] File /usr/lib64/python2.4/site-packages/ZODB/__init__.py, line
31, in ?\nfrom DB import DB
[..] File /usr/lib64/python2.4/site-packages/ZODB/DB.py, line 27, in
?\nfrom ZODB.Connection import Connection
[..] File /usr/lib64/python2.4/site-packages/ZODB/Connection.py,
line 33, in ?\nfrom ZODB.blob import Blob, rename_or_copy_blob
[..] File /usr/lib64/python2.4/site-packages/ZODB/blob.py, line 34,
in ?\nfrom ZODB import utils
[..] ImportError: cannot import name utils
/error1
error2
Traceback (most recent call last):
File /usr/lib64/python2.4/site-packages/mod_python/importer.py, line
1537, in HandlerDispatch
default=default_handler, arg=req, silent=hlist.silent)
File /usr/lib64/python2.4/site-packages/mod_python/importer.py, line
1229, in _process_target
result = _execute_target(config, req, object, arg)
File /usr/lib64/python2.4/site-packages/mod_python/importer.py, line
1128, in _execute_target
result = object(arg)
File /usr/lib64/python2.4/site-packages/mod_python/publisher.py, line
204, in handler
module = page_cache[req]
File /usr/lib64/python2.4/site-packages/mod_python/importer.py, line
1059, in __getitem__
return import_module(req.filename)
File /usr/lib64/python2.4/site-packages/mod_python/importer.py, line
296, in import_module
log, import_path)
File /usr/lib64/python2.4/site-packages/mod_python/importer.py, line
680, in import_module
execfile(file, module.__dict__)
File /opt/indico/www/htdocs/signIn.py, line 22, in ?
from MaKaC.common.general import *
File /usr/lib/python2.4/site-packages/MaKaC/common/__init__.py, line 22,
in ?
from db import DBMgr
File /usr/lib/python2.4/site-packages/MaKaC/common/db.py, line 30, in ?
from ZEO.ClientStorage import ClientStorage
File /usr/lib64/python2.4/site-packages/ZEO/ClientStorage.py, line 34,
in ?
from ZEO.cache import ClientCache
File /usr/lib64/python2.4/site-packages/ZEO/cache.py, line 36, in ?
import ZODB.fsIndex
File /usr/lib64/python2.4/site-packages/ZODB/__init__.py, line 31, in ?
from DB import DB
File /usr/lib64/python2.4/site-packages/ZODB/DB.py, line 27, in ?
from ZODB.Connection import Connection
File /usr/lib64/python2.4/site-packages/ZODB/Connection.py, line 33, in ?
from ZODB.blob import Blob, rename_or_copy_blob
File /usr/lib64/python2.4/site-packages/ZODB/blob.py, line 36, in ?
import transaction
File /usr/lib64/python2.4/site-packages/transaction/__init__.py, line
19, in ?
from transaction._transaction import Transaction
File /usr/lib64/python2.4/site-packages/transaction/_transaction.py,
line 620, in ?
class Savepoint:
File /usr/lib64/python2.4/site-packages/zope/interface/advice.py, line
132, in advise
return callback(newClass)
File /usr/lib64/python2.4/site-packages/zope/interface/declarations.py,
line 484, in _implements_advice
classImplements(cls, *interfaces)
File /usr/lib64/python2.4/site-packages/zope/interface/declarations.py,
line 460, in classImplements
spec = implementedBy(cls)
File /usr/lib64/python2.4/site-packages/zope/interface/declarations.py,
line 300, in implementedByFallback
spec = cls.__dict__.get('__implemented__')
RuntimeError: class.__dict__ not accessible in restricted mode
/error2
Have you ever experience a problem like this? Is there any easy solution?
We use python 2.4.3, apache 2.2.3, mod_python 3.3.1.
Thank you so much in advance.
Best regards,
Jose
Re: [ZODB-Dev] Proposal (version 2): cross database reference seat belt
On 4/28/09 1:54 PM, Jim Fulton wrote: An interim step, if we're in a hurry to get 3.9 out, is to simply add the flag. This would disallow cross-database references in new applications. These applications could still support multiple databases by providing application-level traversal across databases. +1. It seems good to start simple like this. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Database conflict error
ZODB uses an optimistic concurrency model to deal with simultaneous changes to the same object from multiple connections. In practice, this means your software needs to be willing to retry requests when they fail due to one of these errors. Usually this means (in newer ZODBs), aborting the current transaction and starting a new one, replaying the action that caused the ZODB to fail with a conflict error. - C Andrew Thompson wrote: Hi, I've been using the ZODB pretty intensively for a few weeks now, and suddenly have started getting intermittent Database conflict errors, after which calling conn.sync() makes no difference, and I cannot get back to the pre-exception state in one of my ZEO clients, although all the others continue quite happily. I was using 3.7.0a0 and downloaded the 3.8 egg. The behaviour remains the same. I am using a ZEO server with filestorage, and have tried rebuilding the database from scratch, but the intermitent conflict error appears. The error is : database conflict error (oid 0x029b, class btreelist.BTreeQueue, serial this txn started with 0x03750b64c9cf1f00 2008-04-11 08:36:47.299000, serial currently committed 0x03750b64ca57a788 2008-04-11 08:36:47.424000) BTreeQueue is a very simple wrapper over IOBTree. Using integer keys. Question : What does the conflict error actually mean ? Question : All my other ZEO clients continue happily -- and even this process works ***most*** of the time. Short of dropping the ZEO connection and rejoining is there a way of resetting the ZEO session? Regards Andrew Thompson ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: Duplicate tests
Thomas Lotze wrote: Jim Fulton wrote: Chris McDonough did the transaction split off. He's probably the best one to answer your other questions. I know, but then he's subscribed to this list afaik, so I'll just wait for him to respond. I'm afraid I can't look at this right away but I'll put it on the list of things to do. - C If the tests pass without it, then I think it is a safe bet that it can. :) And gone it is. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] transaction 1.0a1 released
FTR, I tagged a release of the new transaction package (http://svn.zope.org/transaction ) as 1.0a1 and uploaded it to PyPi http://pypi.python.org/pypi/transaction/1.0a1) , which is essentially the ZODB transaction machinery divorced from any ZODB package dependencies. The next ZODB release will use a revision of this package. Some non-Zope folks were asking for it and it's useful to get it out there while we wait for a ZODB that depends upon it to get released. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] transaction egg refactoring breaks zodb trunk
Christian fixed these failures, after I hosed things. These were second-level failures. The first level failure was that the tests couldn't be run at all. The way Christian found the failure was that he had a buildbot which couldn't successfully run the tests because the buildbot didn't know enough to attempt to download and put the transaction module in the path during the tests. Phillipp fixed this by adding an SVN external that points to the 'transaction' top-level module within the ZODB SVN module and he changed the buildout.cfg to use this as a 'develop' directory. This works but it's suboptimal for release, because we need to keep track of the fact that this directory shouldn't be shipped in an sdist (and thus add it as an exclude in setup.py). It's currently not excluded, so if we were to make a release today, the release would include the 'transaction' directory. I've made it possible to run the tests now via setup.py test (I just wrap the testrunner) in a clean checkout by adding the appropriate tests_require declarations. I'm not sure how to be able to remove the 'transaction' SVN external while still allowing the tests to pass via buildout's bin/test regime, however. On Nov 10, 2007, at 6:11 AM, Christian Theune wrote: The ZODB trunk is broken after the transaction package was moved: Error in test checkFailingSavepointSticks (ZODB.tests.testZODB.ZODBTests) Traceback (most recent call last): File /home/ctheune/local/python2.4/lib/python2.4/unittest.py, line 260, in run testMethod() File /home/ctheune/tmp/ZODB.trunk/src/ZODB/tests/testZODB.py, line 363, in checkFailingSavepointSticks rt['a'] = 1 File /home/ctheune/tmp/ZODB.trunk/src/persistent/mapping.py, line 53, in __setitem__ self._p_changed = 1 File /home/ctheune/tmp/ZODB.trunk/src/ZODB/Connection.py, line 936, in register self._register(obj) File /home/ctheune/tmp/ZODB.trunk/src/ZODB/Connection.py, line 946, in _register self.transaction_manager.get().join(self) File /home/ctheune/tmp/ZODB.trunk/transaction/transaction/ _transaction.py, line 213, in join self._prior_operation_failed() # doesn't return File /home/ctheune/tmp/ZODB.trunk/transaction/transaction/ _transaction.py, line 207, in _prior_operation_failed raise TransactionFailedError(An operation previously failed, TransactionFailedError: An operation previously failed, with traceback: File bin/test, line 23, in ? zope.testing.testrunner.run([ File /home/ctheune/.eggs/tmprcnM-Q/zope.testing-3.5.1-py2.4.egg/zope/ testing/testrunner.py, line 932, in run File /home/ctheune/.eggs/tmprcnM-Q/zope.testing-3.5.1-py2.4.egg/zope/ testing/testrunner.py, line 1068, in run_with_options File /home/ctheune/.eggs/tmprcnM-Q/zope.testing-3.5.1-py2.4.egg/zope/ testing/testrunner.py, line 1199, in run_tests File /home/ctheune/local/python2.4/lib/python2.4/unittest.py, line 281, in __call__ return self.run(*args, **kwds) File /home/ctheune/local/python2.4/lib/python2.4/unittest.py, line 260, in run testMethod() File /home/ctheune/tmp/ZODB.trunk/src/ZODB/tests/testZODB.py, line 324, in checkFailingCommitSticks self.assertRaises(PoisonedError, transaction.get().commit) File /home/ctheune/local/python2.4/lib/python2.4/unittest.py, line 320, in failUnlessRaises callableObj(*args, **kwargs) File /home/ctheune/tmp/ZODB.trunk/transaction/transaction/ _transaction.py, line 325, in commit t, v, tb = self._saveAndGetCommitishError() File /home/ctheune/tmp/ZODB.trunk/transaction/transaction/ _transaction.py, line 322, in commit self._commitResources() File /home/ctheune/tmp/ZODB.trunk/transaction/transaction/ _transaction.py, line 419, in _commitResources rm.tpc_vote(self) File /home/ctheune/tmp/ZODB.trunk/transaction/transaction/ _transaction.py, line 535, in tpc_vote self.manager.tpc_vote(txn) File /home/ctheune/tmp/ZODB.trunk/src/ZODB/tests/testZODB.py, line 631, in tpc_vote raise PoisonedError(tpc_vote fails) PoisonedError: tpc_vote fails -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] breaking out the transaction module from ZODB
On Nov 9, 2007, at 7:52 AM, Jim Fulton wrote: On Nov 8, 2007, at 7:46 PM, Chris McDonough wrote: How about zope.transaction? Guido recently told me that people in the Python community at large assume that anything in the Zope namespace is assumed to be Zope specific, so I'd rather not put it there. Does it matter? People who are allergic to the name zope can probably lose. It maters to me. OK. I defer here. But in general, I think what would probably work better than a new z namespace or any other avoidance of the zope name is is better dependency specifications, so people would feel better about trying to easy_install zope.* packages. Tres suggested yesterday that we should write a buildbot-like thing that checked out each top-level package in SVN and installed it into a fresh virtualenv to see what its dependencies actually are and fix the too-conservative dependencies. I've made a 'zope.transaction' package that I'll rename to 'transaction': one test still fails in its current state, which I should get fixed today: http://svn.zope.org/zope.transaction/trunk/ WeakSet is in weakset.py. It also contains TimeStamp, which will get moved out of persistent. The tests work (reqt's are downloaded) if you do setup.py test -q - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] breaking out the transaction module from ZODB
On Nov 9, 2007, at 8:41 AM, Jim Fulton wrote: On Nov 9, 2007, at 8:31 AM, Chris McDonough wrote: It also contains TimeStamp, which will get moved out of persistent. Why? I don't see any uses of TimeStamp by the transaction package. In your new package, it is only used by its tests. D'oh! You're right. Out it goes. I wrote tests for it, I'll add them to the persistent package. The tests work (reqt's are downloaded) if you do setup.py test -q Yawn. IMO, the test command in setuptools is a waste of time, because it doesn't work with anything else. It runs all the tests, even the doctests, if thats what you mean. See the additional_tests hair in the test modules. zope.interface is a real requirement. It is already in test_requires. Over time, we need to clean up the transaction tests so they don't use ZODB. Yeah, given that we're name this thing transaction, it's an actually an immediate requirement. There's only one test that uses anything that can't be mocked up in the transaction package (it uses an actual MappingStorage and a DB) itself. It also happens to be the one that fails right now; I haven't tried to understand it yet. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] breaking out the transaction module from ZODB
On Nov 9, 2007, at 9:43 AM, Jim Fulton wrote: Yawn. IMO, the test command in setuptools is a waste of time, because it doesn't work with anything else. It runs all the tests, even the doctests, if thats what you mean. See the additional_tests hair in the test modules. My point is that the meta data you added to the setup.py only works with setup.py. It isn't accessible to any other test runners. I shouldn't have yawned. It is significant that the tests work. :) I just don't find this way of running the tests to be useful. The first time I work on this package, I'll add a buildout.cfg so I can use the Zope test runner. At that point, I'll have to deal with these extra requirements in another way (which is no big deal). What if we caused setup.py to read a buildout.cfg for the tests_require package names and we passed these in as tests_require= names? Would that make it all better? zope.interface is a real requirement. It is already in test_requires. Over time, we need to clean up the transaction tests so they don't use ZODB. Yeah, given that we're name this thing transaction, it's an actually an immediate requirement. If you are going to spend the time, then, uh, sure. ;) Seriously, while I would love to see this cleaned up, I don't think I would consider this super urgent. I guess that depends on competing priorities. I just deleted the sections of the test_transaction doctests that depended on ZODB. They were actually not really testing transactions, they were testing persistent object behavior. I'll try to put them back in a form within ZODB proper, as the test really are testing ZODB functionality, not transaction functionality. There's only one test that uses anything that can't be mocked up in the transaction package (it uses an actual MappingStorage and a DB) itself. It also happens to be the one that fails right now; I haven't tried to understand it yet. Gah. BTW, if you haven't already, you should check for transaction tests lurking in the other ZODB packages. Good idea. In the meantime, I've gotten rid of 'zope.transaction' and I've created a new top-level 'transaction' package at http://svn.zope.org/transaction/ . All its tests pass. It depends only on 'zope.interface', and requires 'zope.testing' for running the tests. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] breaking out the transaction module from ZODB
On Nov 9, 2007, at 12:10 PM, Chris McDonough wrote: BTW, it would be nice to now remove the transaction package from the ZODB trunk and make it a dependency. Yes. ZODB's setup.py is polyglotic... it works if setuptools isn't installed. I suspect it shouldn't continue to given that it now has an external egg dependency. I've made changes to the ZODB setup.py and I've remove the 'transaction' directory from ZODB/src. The changes also imply that setuptools is required to run setup.py, and the 'transaction' distribution is named as an install_requires dependency. Setuptools is now required to install the ZODB head. I made a tag before I did this at http://svn.zope.org/ZODB/tags/before_transaction_remove/ in case I hosed anything in the process. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] breaking out the transaction module from ZODB
On Nov 8, 2007, at 9:14 AM, Jim Fulton wrote: class TransactionFailedError(Exception): Cannot perform an operation on a transaction that previously failed. An attempt was made to commit a transaction, or to join a transaction, but this transaction previously raised an exception during an attempt to commit it. The transaction must be explicitly aborted, either by invoking abort() on the transaction, or begin() on its transaction manager. Why not subclass TransactionError? It didn't before. Should it? There's also a DoomedTransaction exception in the interfaces package that could. TransactionError.__bases__ += (POSError,) TransactionFailedError.__bases__ += (POSError,) Is this *really* necessary? It's obviously a bit evil. Let's explore alternatives to this: 1. Just don't do it. I'd be a bit surprised if there was code actually catching POSError. Me too; +1. If I notice anything relying on them inheriting from POSError, I'll move POSError into the transaction module. - I've created a zc.zodbutils package that is essentially the code that currently lives in the ZODB.utils module; I've also moved the TimeStamp.c code that currently lives in 'persistent' into it. A stub ZODB.utils module exists that just does from zc.zodbutils import *, and in the persistent package's __init__.py, I do from zc.zodbutils import TimeStamp for backwards compatibility. I'd rather not do this. Let's be a bit more selective here. The number of imports from ZODB are pretty limited. Many of them should move to transaction. Some of them are just test utilities that can be duplicated. I think the biggest challenge is WeakSet. This could be broken out into a separate package, but I think it's not as general as its name implies and should probably just be moved to transaction. OK. And I'm thinking that the transaction distribution should be named just transaction. Yes, unless we decide to move the package. I think transaction is a bit presumptuous. :) How about zope.transaction? There's a good deal of 3rd-party code that does import transaction but we could supply a module alias for this purpose. We'd just change the Z2 and Z3 appserver distributions to do import zope.transaction as transaction or whatever, and have the appserver distributions depend on a shim transaction module or module alias or whatever too so 3rd-party code would continue to work, maybe making imports using them issue a deprecation warning? - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] breaking out the transaction module from ZODB
On Nov 8, 2007, at 6:25 PM, Jim Fulton wrote: Why not subclass TransactionError? It didn't before. Should it? Seems logical, but I'm not looking that closely. :) OK. How about zope.transaction? Guido recently told me that people in the Python community at large assume that anything in the Zope namespace is assumed to be Zope specific, so I'd rather not put it there. Does it matter? People who are allergic to the name zope can probably lose. There's a good deal of 3rd-party code that does import transaction Good point. I guess we should leave the package where it is. Note that then we have a tricky issue with avoiding having the package installed twice. I guess we should ignore this for now. :/ It'd be no problem to provide the shims. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HOWTO recover from Data.fs.tmp
On Oct 2, 2006, at 5:38 PM, Juan Pablo Giménez wrote: I'm not blaming the zope team... zope is great! but the real problem wasn't a deleted file... was a corrupted file... and a sysadmin who doesn't read the log file... but... if zope knows about the corrupted Data.fs, why keeps writing into .tmp file?! Saludos... A diagram of that sentence might help: but... if zope knows about the corrupted Data.fs... First of all, do you know that Zope knew the file was corrupt? How do *you* know it had been corrupted? And what does 'corrupt' mean? It seems like even it it had been corrupted in some way (meaning, say, bytes inside of it had been overwritten with garbage, for example), it wouldn't matter, because apparently someone unlinked it so it's not anywhere around to analyze. So I'm afraid nothing can be done without some sort of backup copy or unless Zope is still running and you can get at its file descriptor from within the /proc filesystem or something. why keeps writing into .tmp file?! ... this is not meaningful to the conversation, it'd be better just to leave this part of the sentence above off. In any case, I'm not sure there's anything we can do to help unless you have a backup or unless Zope happens to still be running. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope-Annce] Technical Preview of Blob support in ZODB
On Aug 21, 2006, at 9:13 AM, Christian Theune wrote: David Pratt wrote: Can you advise whether blobs will make it the trunk any time soon. I see an extfile package now for z3 but would rather see filesystem storage dealt with at the backend than in the app itself. Many thanks. Chris McDonough and I have this on our schedule. It's not much to do and I hope to get my todo list done during the next week. After that we might be able to merge into trunk during September. It's not much work if you don't care about Windows, anyway. ;-) We don't have it running at all for Windows yet. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB file never updated - import transaction fails...
Since your exception is happening during __del__ and since AFAIK no
guarantees are made as to what will exist is sys.modules during any
given object's __del__, it's maybe not suprising that you're getting
this exception. I don't know that this has anything to do with ZODB
in particular.
The first thing I'd try to do is to move the cleanup code out of
__del__ (and perhaps to a finally: after the line that creates the
cache instead) and see if that helps.
- C
On Jul 29, 2006, at 9:40 AM, [EMAIL PROTECTED] wrote:
Matt Cowles has a DNS cache module with a patch for use with
SpamBayes. It
doesn't persist the data (just caches during the lifetime of the
run). I'm
trying to add persistence to that cache and wanting to use the same
database
as the user selects for the main SpamBayes database, my first two
targets
were anydbm and ZODB. I'm having trouble with the ZODB setup. It
seemed to
work well at the start, but the database file is never updated when
I rerun
my training script, so its not caching anything, and now everything
in the
database has expired. I've never used ZODB before. The amount of
code is
fairly small, so I'll just post it here in hopes someone can see
what I've
done wrong.
When opening the database it selects based on the user-preferred
filetype.
For zodb it executes:
from ZODB import DB
from ZODB.FileStorage import FileStorage
self._zodb_storage = FileStorage(cachefile, read_only=False)
self._DB = DB(self._zodb_storage, cache_size=1)
self._conn = self._DB.open()
root = self._conn.root()
self.caches = root.get(dnscache)
if self.caches is None:
# There is no classifier, so create one.
from BTrees.OOBTree import OOBTree
self.caches = root[dnscache] = OOBTree()
self.caches[A] = {}
self.caches[PTR] = {}
That gives the cache a caches attribute that looks like the
dictionary it
uses in the no persistence case.
When the cache is deleted, its __del__ method calls
self.close()
which calls self._zodb_close():
def _zodb_close(self):
# Ensure that the db is saved before closing. Alternatively, we
# could abort any waiting transaction. We need to do
*something*
# with it, though, or it will be still around after the db is
# closed and cause problems. For now, saving seems to make
sense
# (and we can always add abort methods if they are ever needed).
self._zodb_store()
# Do the closing.
self._DB.close()
which calls self._zodb_store():
def _zodb_store(self):
import transaction
from ZODB.POSException import ConflictError
from ZODB.POSException import TransactionFailedError
try:
transaction.commit()
except ConflictError, msg:
# We'll save it next time, or on close. It'll be lost if we
# hard-crash, but that's unlikely, and not a particularly
big
# deal.
if options[globals, verbose]:
print sys.stderr, Conflict on commit., msg
transaction.abort()
except TransactionFailedError, msg:
# Saving isn't working. Try to abort, but chances are that
# restarting is needed.
if options[globals, verbose]:
print sys.stderr, Store failed. Need to restart.,
msg
transaction.abort()
which seems to fail. I get this message in the log:
Exception exceptions.ImportError: 'No module named transaction' in
bound method cache.__del__ of spambayes.dnscache.cache
instance at
0x11994b8 ignored
I can import transaction from the interpreter prompt just fine:
% python
Python 2.5b2 (trunk:50921, Jul 28 2006, 20:21:50)
[GCC 4.0.0 (Apple Computer, Inc. build 5026)] on darwin
Type help, copyright, credits or license for more
information.
import transaction
transaction.__file__
'/Users/skip/local/lib/python2.5/site-packages/transaction/
__init__.pyc'
import ZODB
ZODB.__file__
'/Users/skip/local/lib/python2.5/site-packages/ZODB/__init__.pyc'
I stuck a
print transaction.__file__
after the import in _zodb_close and ran dnscache as a main. It
prints out
/Users/skip/local/lib/python2.5/site-packages/transaction/
__init__.pyc
so I'm pretty sure I'm getting the same version of Python. Any
idea why it
would fail in one instance but not another? The only thing I can
think of
is that there are two ZODB databases opened at the same time in the
real use
case, but only the one in the situation where I'm running the
dnscache test
function. Is there something about having two distinct ZODB
database files
open I need to consider?
Thanks,
--
Skip Montanaro - [EMAIL PROTECTED] - http://www.mojam.com/
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
Re: [ZODB-Dev] tempstorage to ZEO?
Fine by me. - C On May 31, 2006, at 1:22 AM, Sidnei da Silva wrote: I was looking, and trying to understand why 'tempstorage' is not included in the ZODB. Since some people might want to share sessions between ZEO clients, and that 'tempstorage' is what is mounted as '/temp_folder/session_data', why not make that easy to mount via ZEO? Does anyone see a problem with this? What I'm proposing is to move 'tempstorage' to 'ZODB', since it has no dependency on anything other than 'zLOG'. -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] tempstorage to ZEO?
On May 31, 2006, at 9:08 AM, Sidnei da Silva wrote: On Wed, May 31, 2006 at 09:53:14AM +0200, Tino Wildenhain wrote: | Sidnei da Silva schrieb: | I was looking, and trying to understand why 'tempstorage' is not | included in the ZODB. | | Since some people might want to share sessions between ZEO clients, | and that 'tempstorage' is what is mounted as | '/temp_folder/session_data', why not make that easy to mount via ZEO? | Does anyone see a problem with this? | | actually it is easy mountable via ZEO. I'm doing this already... | Do you mean including the schema for zeo.conf per default? Yeah, that too. I was actually thinking about the standalone ZODB package. Or does that not exist anymore? I was also wondering if this is the right thing to do (mounting tempstorage via ZEO) since I don't recall seeing any tutorial that recommended it. Should be OK, although if you're doing it because you want a fast storage (or one that produces fewer conflicts), ZEO takes away much hope of that intrinsically. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] tempstorage to ZEO?
Sorry, I've not run your tests but I'll note that they don't take into account any concurrency or time lost due to conflict error retries. There are many more conflicts when writing to ZEO-backed database in general when there is lots of concurrency because write transactions usually take longer. And even if MVCC bails you out, it still takes time to do the conflict resolution. There is a set of tests explicitly created to torture test session conflict rates here: http://cvs.zope.org/Packages/SessionRig/ See also: http://www.plope.com/Members/dunny/conflicts/view - C On May 31, 2006, at 10:58 PM, Sidnei da Silva wrote: On Wed, May 31, 2006 at 09:59:36AM -0400, Chris McDonough wrote: | | On May 31, 2006, at 9:55 AM, Sidnei da Silva wrote: | | On Wed, May 31, 2006 at 09:49:49AM -0400, Chris McDonough wrote: | | I've done this (at least with FileStorage) and it's | | sllo. Might be OK for low-traffic sites, but better | | to implement a custom session data container that stores stuff in | | SQL. I have the beginnings of one of these if you want to see it. | | Slow for using sessions? Why didn't you use tempstorage then? | | I think I thought it wouldn't have mattered. The difference between | using sessions against a local filestorage and one on a ZEO server | was something like 20X. So, since I couldn't believe the 20X figure, I wrote a very dumb test [1] that shows [2] tempstorage to be only slightly slower than filestorage (roughly 10%) and that adding zeo to the mix makes both of them about 4X slower, which would be pretty acceptable by my standards. I'm looking forward for testing tres' memcached stuff tomorrow. In the meantime, it would be great if someone can run the script on different boxes and platforms (I've ran it on Windows on a Intel Dual Core 3.0) to see if there's any difference. [1] http://awkly.org/files/zeo-bench/bench.py [2] http://awkly.org/files/zeo-bench/bench_results.txt -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zodb with Yaml
Hmm. A scan of the files in ZODB and persistence also turn up the conflict resolution code, the import/export machinery, and FileStorage. And a bunch of tests. And maybe the pickle cache and cPersistence.c (although I haven't looked closely enough at that code to know if it would need to be modified, maybe the function names in the code are misleading). - C On Mar 19, 2006, at 2:37 PM, Dieter Maurer wrote: Chris McDonough wrote at 2006-3-18 20:58 -0500: If yaml (or pysyck) supports all of the features of pickle, it's theoretically possible. You would need to track down all of the places in ZODB where the code serializes and deserializes data from pickle format and write alternate serializing/deserializing logic. But AFAIK the serialization code in ZODB is not very pluggable. I am much more optimistic. I think only ZODB/serialize.py and ZODB/Connection.py would need to be modified. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Mar 3, 2006, at 8:08 AM, Sidnei da Silva wrote: On Thu, Mar 02, 2006 at 10:03:48PM -0500, Chris McDonough wrote: | I'm taking a stab at packaging the various pieces of ZODB as eggs. | One of the things I'd like to do is to separate the packages of ZODB | that are currently shipped together (BTrees, ZODB/ZEO, persistent | transaction, ZConfig) into separate packages, because each may | arguably be useful outside of what we call ZODB now. | | One of the issues I've run into is that there are compile-time | dependencies between packages mainly due to C header files. For | example, the BTrees package assumes that it will be able to find the | cPersistence.h file, which currently ships with the persistent | package. I wonder if I should bother breaking the persistent and | BTrees packages apart as separate eggs. I suspect not. Can't you make them separate eggs that depend on the other? Well, yes. I've already done that. But it's a hack. The runtime dependencies aren't a problem, it's the build-time dependencies. I don't know of a way to say use this header file from this egg to build this other egg in an egg setup script. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Mar 3, 2006, at 10:13 AM, Sidnei da Silva wrote: | Well, yes. I've already done that. But it's a hack. The runtime | dependencies aren't a problem, it's the build-time dependencies. I | don't know of a way to say use this header file from this egg to | build this other egg in an egg setup script. Maybe it's a missing feature. I would ask Phillip Eby. Maybe it is, but in this case, is it worth it? Does anyone want to use BTrees outside of a persistent hierarchy? - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Mar 3, 2006, at 11:46 AM, Jeremy Hylton wrote: Go go! I don't understand the eggs philosophy. (I succeeded in missing the eggs talk twice at PyCon.) I don't think any of the pieces of ZODB are very useful in isolation. You can't use persistent without transaction, and there isn't much that uses transaction other than ZODB. Same with BTrees, you can't use them without ZODB and they're probably the most useful data structure in ZODB. I think you put transaction in a separate package because it was meant to be useful independent of ZODB. There's no actual code in persistent that depends on transaction save for the tests (although I couldn't imagine using persistent without transaction, is there ever a case?). Maybe the right packaging division is transaction, ZConfig, and everything else? - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] packaging zodb in eggs.
On Mar 3, 2006, at 11:47 AM, Jim Fulton wrote: BTW, the turbogears folks are interested in using transaction. The transaction package has shallow dependencies on ZODB. A nice start would be to release a separate transaction egg that doesn't depend on ZODB. (Hint, we'll need a separate weak-set egg that transaction and ZODB depend on and we'll need to decouple the transaction exceptions from ZODB.) OK, that sounds like a place to start, thanks. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)
On Feb 2, 2006, at 3:11 AM, Chris Withers wrote: Chris McDonough wrote: Pound. *barf* ew.. Works great. Except it doesn't balance load and can't do SSL correctly, right? ;-) I use Apache for the SSL and it balances load well enough for me. Can we stop talking now? ;-) - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)
On Feb 1, 2006, at 11:02 AM, Chris Withers wrote: Hi Chris, Chris McDonough wrote: Rather than chucking this in the collector as-is, it would be useful if you could get me the output of the ZEO log rather than the output of the client log. Right now, the actual traceback is in the ZEO log; the client log only shows that a conflict error happened; not where it happened. Your wish is my command: snip Thanks. FWIW, I've not tested ZEO+tempstorage, as in my own deployments I always use session affinity which always sends subsequent requests to the same appserver that received a request containing a particular session cookie. What front end do you use to do the request distribution? Pound. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)
On Feb 1, 2006, at 11:12 AM, Chris Withers wrote: Chris McDonough wrote: What front end do you use to do the request distribution? Pound. *barf* ew.. Works great. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird ConflictErrors with Zope 2.8.4 - It's been a while ; -)
Rather than chucking this in the collector as-is, it would be useful if you could get me the output of the ZEO log rather than the output of the client log. Right now, the actual traceback is in the ZEO log; the client log only shows that a conflict error happened; not where it happened. FWIW, I've not tested ZEO+tempstorage, as in my own deployments I always use session affinity which always sends subsequent requests to the same appserver that received a request containing a particular session cookie. Thanks! - C On Jan 24, 2006, at 8:41 AM, Chris Withers wrote: Okay, now that we have 2.8.4 in place, we get proper reporting of ConflictErrors and today we started seeing one happening over and over again which looked roughly as follows: Traceback (most recent call last): File lib/python/Products/Transience/Transience.py, line 844, in new_or_existing item = self.get(key, _marker) File lib/python/Products/Transience/Transience.py, line 491, in get item = self._move_item(k, current_ts, default) File lib/python/Products/Transience/Transience.py, line 346, in _move_item if getattr(self._data[current_ts][k], 'setLastAccessed', None): File lib/python/ZODB/Connection.py, line 704, in setstate self._setstate(obj) File lib/python/ZODB/Connection.py, line 760, in _setstate self._reader.setGhostState(obj, p) File lib/python/ZODB/serialize.py, line 495, in setGhostState state = self.getState(pickle) File lib/python/ZODB/serialize.py, line 488, in getState return unpickler.load() File lib/python/ZODB/serialize.py, line 436, in _persistent_load return self._conn.get(oid) File lib/python/ZODB/Connection.py, line 207, in get p, serial = self._storage.load(oid, self._version) File lib/python/ZEO/ClientStorage.py, line 746, in load return self.loadEx(oid, version)[:2] File lib/python/ZEO/ClientStorage.py, line 769, in loadEx data, tid, ver = self._server.loadEx(oid, version) File lib/python/ZEO/ServerStub.py, line 192, in loadEx return self.rpc.call(loadEx, oid, version) File lib/python/ZEO/zrpc/connection.py, line 536, in call raise inst # error raised by server ConflictError: database conflict error (oid 0x010f0d) The things which got me worried is that it was always the same oid that was conflicting, and that's not usually something we see. We were also getting it very frequently, and on both reads and writes. That in itself is enough to get me to mail here. I had a dig on the ZEO server and found the matching traceback for one of them there: -- 2006-01-24T12:42:57 INFO ZEO.zrpc.Connection(S) (x.x.x.x:y) loadEx() raised exception: database conflict error (oid 0x010f0d) Traceback (most recent call last): File lib/python/ZEO/zrpc/connection.py,line 421, in handle_request ret = meth(*args) File lib/python/ZEO/StorageServer.py,line 248, in loadEx return self.storage.loadEx(oid, version) Filelib/python/tempstorage/TemporaryStorage.py, line 132, in loadEx data = self.load(oid, version) Filelib/python/tempstorage/TemporaryStorage.py, line 117, in load raise POSException.ConflictError(oid=oid) ConflictError: database conflict error (oid 0x010f0d) Now, I wanted to find out what the troublesome object behind 0x010f0d was, so I fired up and debug zope client, and was distrubed by the result: app.temp_folder._p_jar[p64(0x010f0d)] Traceback (most recent call last): File stdin, line 1, in ? File lib/python/ZODB/Connection.py, line 207, in get p, serial = self._storage.load(oid, self._version) File lib/python/ZEO/ClientStorage.py, line 746, in load return self.loadEx(oid, version)[:2] File lib/python/ZEO/ClientStorage.py, line 769, in loadEx data, tid, ver = self._server.loadEx(oid, version) File lib/python/ZEO/ServerStub.py, line 192, in loadEx return self.rpc.call(loadEx, oid, version) File lib/python/ZEO/zrpc/connection.py, line 536, in call raise inst # error raised by server ZODB.POSException.ConflictError: database conflict error (oid 0x010f0d) We managed to stop the problem causing furthe conflict errors by fiddling with the timeout resolution on the Transient Object Container at /temp_folder/session_data through the web. I guess this re-created the data structures used by the temp_folder or session_data objects and so took the object with the dodgy oid out of the equation. However, even now, an hour or so later, if I fire up a debug client and try and look up that oid, I get a ConflictError. Any idea what's going on? cheers, Chris ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list -
Re: [ZODB-Dev] RE: lost implementer
Package depth has no impact on performance (at least that I've noticed). It's almost certainly not the primary reason for any given Python program to be slow. I'd like to read that blog entry to see if that person knows more than I do. As far as eggs go, AFAIK, package depth is not really an interesting metric either. It seems immaterial how deep an egg package is. - C On Jan 23, 2006, at 10:50 AM, Tamas Hegedus wrote: Hi! Somewhere I have red (in a blog) comparing java and python. If I remember well the main reason to write that was the Boa (?) constructor written by java programmers... So: It states that Python packages should be as flat as possible (use less names in the namespace (?)), since the name space is resolved during running time not at compilation time in case of Python (or something like that; I am not a programmer). So if you have a deeply structured package (or more packages) you may suffer in performance at runtime. (Boa is very slow; although it is use also a lot of 'reflection'.) What are you comments? Do you have some experience with performance problem using eggs (or very 'deep' packages? -- My point of view: it would be great to have ZODB, ZOE, ZCatalog in different eggs :-) Tamas Chris McDonough wrote: See the egg intro doc at http://peak.telecommunity.com/DevCenter/ PythonEggs . -- Tamas Hegedus, PhD | phone: (1) 919-966 0329 UNC - Biochem Biophys | fax: (1) 919-966 5178 5007A Thurston-Bowles Bldg | mailto:[EMAIL PROTECTED] Chapel Hill, NC, 27599-7248 | http://biohegedus.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: session problems
I suppose folks includes me. ;-) I'll try to have a look this week... Sorry, I've been avoiding any actual work over the last week or so. - C On Jan 1, 2006, at 10:23 PM, Florent Guillaume wrote: Could folks have a look and tell me what they think of the proposed patch? Actually it is my belief this is a concern of DB or Connection, not MountedObject, and the code shouldn't be in MountedObject but that's the easiest workaround for now. Florent Florent Guillaume wrote: Ok I've dug deeper and now understand the problem. The root cause is in the multi-databases support. The problem is that the Zope startup only closes the main connection it had on the root database. The first connection to the TemporaryStorage, created and opened during Zope startup, is never closed, so still is a synchronizer in its original transaction, but is nevertheless reused in other transactins, without a proper synchronizer set up. When a MountedObject needs to be traversed, it tries to find an existing connection for the new database by doing: conn = anyjar.get_connection(db_name) where anyjar is the parent connection. If there's a linked connection for that database, it's returned, otherwise if the multi- database already has seen the wanted database, it opens a connection from it, then adds it to the linked connections attribute (conn.connections) and shares this attribute between the two connections. I that fails, because the connection has never been linked to the new database (which is the case during startup code), then the MountedObject code does: conn = self._getDB().open() Here _getDB() correctly returns a newly instanciated database, which has been linked to the other ones in the multi-databases setup (shared databases dictionnary attribute, ultimately coming from Zope2.Startup.datatypes.DBTab.databases). Then open() returns a new opened connection for that database. *BUT* this new connection is not linked to the others (using their .connections attribute). This code from get_connections is needed: self.connections.update(new_con.connections) new_con.connections = self.connections which would be written, in the context of code executing in MountedObject (in _getMountedConnection): except KeyError: conn = self._getDB().open() anyjar.connections.update(conn.connections) conn.connections = anyjar.connections return conn But of course really this code doesn't belong to MountedObject. This is just the simplest way I could find, if others want to test it. The .connections sharing is really funky, apparently all the connections opened in the context of the same multi-databases support are intended to be present in it. Why is this access not indirected through the multi-databases support in DB itself? Also I don't understand why open()'s delegate attribute is not stored as a connection attribute, and close() should reuse it instead of obeying a primary attribute. Anyway, I guess historical code, etc. I'll let specialistst of the multi-databases decide what to do :) Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] In search of Information for ZODB University Paper
Hi Marcus, There is a presentation about ZODB by Jim Fulton at http://www.python.org/workshops/2000-01/proceedings/papers/fulton/zodb3.html . This is probably one of the earliest and most in-depth presentations about the subject. http://www.zope.org/Wikis/ZODB/FrontPage is wildly useful for detailed information about specific ZODB features and issues. I created slightly less abstract slides for a presentation about ZODB a few months ago that you can get here: http://www.plope.com/static/presentations/zodb.pdf (although there is at least one factual errors in there now: Zope 2.9/3.2 will ship with ZODB 3.6, not 3.5). - C On Wed, 2005-10-26 at 20:59 +0200, Markus Kalb wrote: Hi there i'm in search for some general information about ZODB, to help me compile a presentation about ZODB. With google if found a lot of information (maybe even to much) and so i thought i would ask you people if you could point me to some resources ? At the moment i only now ZODB the Plone/Zope way and would like to dive a little deeper into it. (that the reason why i asked to cover ZODB in the class in the university [www.fbi.fh-darmstadt.de]). Thanx in advance for any help ! ciao markus kalb -- .- .-. -.- ..- ... / -.- .- .-.. -... GPG 3E82B98B [EMAIL PROTECTED] ICQ 202340367 -- .- .-. -.- ..- ... / -.- .- .-.. -... ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Blobs revisited...
FYI, Last week, Christian Theune and I muscled the elderly ctheune- blobsupport-branch of ZODB into shape to work again against the ZODB trunk. We ran into some interesting issues with supporting savepoints (difficult to do efficiently with blobs, so we didn't try), I added some somewhat suspect code to support blobs using a TmpStore, and we as always need to write some more tests, but it works and the result is on the blob-merge-branch of ZODB in SVN. We also created a Zope branch to handle the ZODB head. Namely, we used multidatabase support. This is in the zodb-blobs-branch of Zope in SVN. This basically consisted of ripping out old DBTab code and replacing it with calls into ZODB's multidatabase support. It also works. I'm still working on this in my downtime. If anyone else is interested in helping out, let me know! - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Reloading product in 2.8
Refresh was never much of a timesaver for me. Pressing ctrl-C - up arrow - return on the console from which I run Zope is my answer. I know lots of people say that this is too slow, but on my most recent project from the time I type runzope to the time I see ready to handle requests is real0m3.029s user0m1.857s sys 0m0.306s ... and I can live with that. - C On Thu, 2005-07-14 at 23:02 +0200, Dieter Maurer wrote: Lennart Regebro wrote at 2005-7-12 20:22 +0200: ... well, if it happens to work for you, then good luck, but it is actually a piece of unsupported crap that hasn't worked for years and that nobody is bothered to fix bugs in. I have the feeling that we have an easy tendency to say such things: Let a bug occur in some component and then, instead (or in addition) of fixing the bug, we say often: rip the component off Zope. That was threadened for Versions, and ZClasses and now for Refresh. For me, Refresh always was a big development time saver -- I knew that I had to refresh with a product also all dependent products. Recently, I automated this refreshing of dependent products. This made Refresh behave as expected (provided the product dependancies are described correctly) ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: Re: [ZODB-Dev] Handling more databases with zeo
I also use this feature but I would be happy to stop using it if it made the code materially simpler to maintain. - C On Wed, 2005-06-29 at 15:31 -0400, Jeremy Hylton wrote: On 6/29/05, Tim Peters [EMAIL PROTECTED] wrote: [Tim Peters] As before, I'd run a different ZEO server for each database. I'm not sure that what you're doing here will be supported for much longer (or really even _is_ supported anymore -- see my last msg). [Dieter Maurer] I do not know whether it is supported but it works in ZODB 3.2. Why do you want to cancel this? I didn't say I would cancel it / rip it out. I said it's undocumented, untested, and that its status is unclear; and I quoted a comment from current ZODB source that strongly seemed to imply its author (probably Jeremy) believed it was already dead meat (This argument is primarily for backwards compatibility with servers that supported multiple storages -- why did the comment use past tense if the current code still supports multiple storages? I used the past tense because I thought we had decided to cancel the feature at some point. The feature itself has been around and undocumented for much longer. The reason I want to remove the feature is that it adds complexity to the software and configuration without providing much real benefit. The benefit is that you get to run several storages using a single ZEO server process and TCP port. It's probably not a good idea to use a single process for many servers, although it might be convenient to use a single port. Reasons to get rid of it (recalling these from the distant past): - People were confused about what the feature actually did. I helped people several people debug problems that were caused by confusion around this feature. - It's probably better to run separate ZEO processes (possibly on different machines). - There would be less code to maintain and few features to test. Jeremy ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Generational Caching
One way to potentially soften the impact of cache busting by spiders might be to allow Zope to choose a particular ZODB connection based on request parameters (like sessionid or requesting ip address, or most likely user agent in the case of legitimate spiders). This is a modification to Zope that would be largely independent of ZODB, however, so might be more appropriate to discuss on zope-dev. On Wed, 2005-06-08 at 10:20 +1000, Dylan Jay wrote: I've been thinking about the problems of memory use. One use case that seems hard is web crawlers. That will wake up a lot of objects for just a single read possibly getting rid of many often used objects from the zodb cache. Has anyone investigated the idea of a generational cache? Another idea I had was asymetric cache sizes. I believe that zope uses a single thread if all requests are serial and just uses the other connections when similtanious requests are handled. Since some threads will be less used than others perhaps there could be cache settings so memory can be minimised. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
