Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote: My scenario is akin a consumer-producer with shared buffer. Consumers pull items from the buffer whilst producers put items in the buffer. The buffer is an OOBTree along with an IOBTree which gives serial numbers to the keys of the OOBTree. I'm not sure if this is a match, but if you follow Chris' suggestions (which I recommend generally) then you might want to look at zc.queue: http://cheeseshop.python.org/pypi/zc.queue/ . Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
Hi Gary, Although the shared buffer indeed has a queue-like interface there's a catch for my needs. I need that pull method to retrieve the item in the head and to move the head forward, and not to remove the item from the buffer. Just like zc.queue I need a single copy of a item in the queue, but I also need this uniqueness over the time, so that any item which has being consumed already, does not get back to buffer. Thanks for your advice, I haven't seen zc.queue, so you comments are really an improvement to my knowledge. Besides I'm going to take a peek at zc.queue and compare it with my own implementation. Thanks and best regards, Manuel. Gary Poster wrote: On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote: My scenario is akin a consumer-producer with shared buffer. Consumers pull items from the buffer whilst producers put items in the buffer. The buffer is an OOBTree along with an IOBTree which gives serial numbers to the keys of the OOBTree. I'm not sure if this is a match, but if you follow Chris' suggestions (which I recommend generally) then you might want to look at zc.queue: http://cheeseshop.python.org/pypi/zc.queue/ . Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Feb 12, 2007, at 10:01 AM, Manuel Vazquez Acosta wrote: Hi Gary, Although the shared buffer indeed has a queue-like interface there's a catch for my needs. I need that pull method to retrieve the item in the head and to move the head forward, and not to remove the item from the buffer. Just like zc.queue I need a single copy of a item in the queue, but I also need this uniqueness over the time, so that any item which has being consumed already, does not get back to buffer. Ah, OK. FWIW, you could get the same functionality with a wrapper around a zc.queue pretty easily, of course, with at least a couple of variants off the top of my head (both involving an additional data structure that you maintain, and supplying your own `put` and `pull` that wrap the zc.queue methods, either with composition or inheritance). If, instead, you move to a multi-threaded model following the same approach you outlined before (sequential int keys, IIUC) you'll need to be careful of write conflict errors. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
On Sun, Feb 11, 2007 at 07:29:58PM -0500, Manuel Vazquez Acosta wrote: The other question is about compiling ZODB without using the out-of-the-box distutils installation procedure. I'm also playing with Zope and Plone, so I have several instances on the same machine. I think installing with distutils may cause conflicts with the Zope instances. Am I right? If so, then how should I install ZODB side-by-side the consumer-producer application? This sounds like a job for zc.buildout. Marius Gedminas -- Special bonus feature: absolutely nowhere in RetchMail's code is there an arbitrary 3-second sleep(). Wow! What other mail retriever can say that? (Hint: not fetchmail.) -- http://open.nit.ca/wiki/index.php?page=RetchMail signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] Storage error with big transactions.
On Feb 12, 2007, at 12:25 PM, Andreas Jung wrote: I have the following script to emulate a long running writing ZEO client by writing 100MB to a page template: import transaction pt = app.foo while 1: data = '*'*1 T = transaction.begin() pt.pt_edit(data, 'text/html') T.commit() print 'done' This script fails badly during during the first commit() call. Is this a bug or feature? I am using Zope 2.10.2 on MacOSX Intel. Based on the traceback you gave, this looks like a bug. I've noticed, however, that large database records can lead to memory errors at sizes much smaller than I would expect. If the problem is ultimately traced to a hidden memory error, there's not much that can be done. In the long run, I expect we'll advise that large objects be put in blobs, where large might be smaller than one might expect. For example, I've seen 90MB records lead to memory errors even on machines with a hundreds of megabytes free. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [ZEO] Storage error with big transactions.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: On Feb 12, 2007, at 12:25 PM, Andreas Jung wrote: I have the following script to emulate a long running writing ZEO client by writing 100MB to a page template: import transaction pt = app.foo while 1: data = '*'*1 T = transaction.begin() pt.pt_edit(data, 'text/html') T.commit() print 'done' This script fails badly during during the first commit() call. Is this a bug or feature? I am using Zope 2.10.2 on MacOSX Intel. Based on the traceback you gave, this looks like a bug. I've noticed, however, that large database records can lead to memory errors at sizes much smaller than I would expect. If the problem is ultimately traced to a hidden memory error, there's not much that can be done. In the long run, I expect we'll advise that large objects be put in blobs, where large might be smaller than one might expect. For example, I've seen 90MB records lead to memory errors even on machines with a hundreds of megabytes free. That sounds like a fragmented heap. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF0f0I+gerLs4ltQ4RArFgAKCB0dTHcLLgoZ1ZAAq6XKbCaizAZgCfRFgT iuCdpw99+dU/cwExlTPdTJ4= =coHy -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Making use of the zodb transaction framework outside of zodb?
Hello all, In the docs I have read that it is possible for non-zodb apps to plug into the transaction framework. However, I am unable to find any specifics as to how to do this. What I'd like to do is this: 1. Have my app import transaction 2. When transaction.commit() is called from my app, have other things be notified by that. I'm a bit of a newbie so I may have missed something very obvious in the docs. Is anyone able to tell me either how to do write this, or where in the docs has specific examples? Many thanks, PC Peloton Partners LLP is registered in England (registration number OC307362) with its registered office address at 17 Broadwick Street, London W1F 0DJ. This message may contain confidential and/or privileged material. It is intended only for use by the named recipient(s). This message is for informational purposes only and must not be construed as an offer or solicitation to buy or sell any security. Peloton Partners LLP does not represent that it is accurate, complete and/or up-to date and accepts no liability if it is not. All opinions expressed do not necessarily reflect those of Peloton Partners LLP or its affiliates. Peloton Partners LLP is authorised and regulated by the Financial Services Authority. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] Storage error with big transactions.
[Andreas Jung] I have the following script to emulate a long running writing ZEO client by writing 100MB to a page template: import transaction pt = app.foo while 1: data = '*'*1 T = transaction.begin() pt.pt_edit(data, 'text/html') T.commit() print 'done' This script fails badly during during the first commit() call. Is this a bug or feature? I am using Zope 2.10.2 on MacOSX Intel. Probably a bug: ... suxmac2:~/sandboxes/zeo-test ajung$ bin/zopectl run test.py 2007-02-12 18:18:18 CRITICAL txn.-1610559552 A storage error occurred during the second phase of the two-phase commit. Resources may be in an inconsistent state. Traceback (most recent call last): File string, line 1, in ? File test.py, line 8, in ? T.commit() File /opt/zope/2.10.2/lib/python/transaction/_transaction.py, line 395, in commit self._commitResources() File /opt/zope/2.10.2/lib/python/transaction/_transaction.py, line 503, in _commitResources rm.tpc_finish(self) File /opt/zope/2.10.2/lib/python/ZODB/Connection.py, line 696, in tpc_finish self._storage.tpc_finish(transaction, callback) File /opt/zope/2.10.2/lib/python/ZEO/ClientStorage.py, line 955, in tpc_finish self._update_cache(tid) File /opt/zope/2.10.2/lib/python/ZEO/ClientStorage.py, line 980, in _update_cache self._cache.invalidate(oid, version, tid) File /opt/zope/2.10.2/lib/python/ZEO/cache.py, line 375, in invalidate assert o is not None AssertionError Under the covers someone is trying to invalidate a current ZEO cache entry, and the ZEO cache is complaining because it doesn't believe it /has/ current data (assert o is not None). Best guess for why it doesn't have it is that it silently refused to add current data to begin with, because the object pickle was bigger than the size of the disk file allocated for the ZEO cache. Offhand I don't recall that this situation is tested, so it's likely to fail in some way. One way to check: configure the ZEO cache file to be reasonably bigger than the size of `data = '*'*1`. If the problem goes away then, that's the cause ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
Manuel Vazquez Acosta wrote at 2007-2-11 19:29 -0500: ... I have read that each thread should have its own connection to the DB. However, in my case, each thread should be aware of what is actually in the DB at all times. So I wonder if I can shared the connection between those threads as long as I take the means to protect it (i.e RLock). You might be able to do that -- but you would loose almost all benefits of the ZODB: an earlier discussion with Jim and Jeremy indicated that not only writes would need such locking but reads as well I think, you should forget about using the same ZODB connection in different threads... -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [ZEO] Storage error with big transactions.
Andreas Jung wrote at 2007-2-12 18:25 +0100: ... This script fails badly during during the first commit() call. Is this a bug or feature? I am using Zope 2.10.2 on MacOSX Intel. Looks like a bug. Apparently, an invalidation message arrives at the ZEO client cache for an object that is not in the cache. Of course, this is a standard case and must be handled correctly by the cache. However in your (buggy) case, the object is not in the FileCache (on disk) instance but self.current contains the object. This inconsistency between the on disk state and the in memory state causes the bug. Looks like a ZEO.cache.ClientCache bug. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
