Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
Chris, Thanks for your response. I will try the locking first and the see if I can implement a transaction manager of my own. I will read about virtualizing Python, I'm a debian user :)) Thanks and best regards, Manuel. Chris McDonough wrote: On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote: Hi all, I'm writing a small application for which I want to use ZODB's persistence machinery. I have a couple of question, though. I have read that each thread should have its own connection to the DB. However, in my case, each thread should be aware of what is actually in the DB at all times. So I wonder if I can shared the connection between those threads as long as I take the means to protect it (i.e RLock). I am not certain, actually. The mechanism for doing this is in there. The interfaces.py file states: Synchronization --- A Connection instance is not thread-safe. It is designed to support a thread model where each thread has its own transaction. If an application has more than one thread that uses the connection or the transaction the connection is registered with, the application should provide locking. And it goes on to talk about the sync=False parameter vaguely. But the story doesnt really end there... there is also the capability to register a different transaction manager at DB open time.. the default one assumes one transaction per thread. I'd need to dig through all this code to reunderstand it, but it looks possible. My scenario is akin a consumer-producer with shared buffer. Consumers pull items from the buffer whilst producers put items in the buffer. The buffer is an OOBTree along with an IOBTree which gives serial numbers to the keys of the OOBTree. The typical way to do this using one-connection-per-thread and a threaded txn manager would be to call transaction.commit() when mutating the shared data structures. To detect changes to the objects provided by a connection (due to other threads committing during the connection's transaction), you can connection.sync(). Zope calls transaction.begin() at the beginning of a request and either transaction.abort() (on failure) or transaction.commit() (on success) at its termination. It never needs to do an explict connection.sync() because it never wants to do a dirty read (each request has its own connection and sees its connection data as canonical for the duration of its existence). Zope handles conflicts by retrying the request. But Zope has it easy because the web has natural transaction boundaries that don't always exist in GUI apps or other apps with a different request/response cycle. The other question is about compiling ZODB without using the out-of-the-box distutils installation procedure. I'm also playing with Zope and Plone, so I have several instances on the same machine. I think installing with distutils may cause conflicts with the Zope instances. Am I right? If so, then how should I install ZODB side-by-side the consumer-producer application? The easiest thing to do here is to install a separate Python instance for each project. Alternately if you're on UNIX, you can use virtual python (http://peak.telecommunity.com/dist/virtual-python.py) which makes a tree of symlinks that acts like a new Python instance but has its own site packages directory in which distutils stuff is installed. If you're on Windows, I believe Zope/Plone ship with their own copy of Python on that platform, so installing to the system python shouldnt create a conflict. - C ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Threads, ZODB, and how to install ZODB without distutils
Hi Gary, Although the shared buffer indeed has a queue-like interface there's a catch for my needs. I need that pull method to retrieve the item in the head and to move the head forward, and not to remove the item from the buffer. Just like zc.queue I need a single copy of a item in the queue, but I also need this uniqueness over the time, so that any item which has being consumed already, does not get back to buffer. Thanks for your advice, I haven't seen zc.queue, so you comments are really an improvement to my knowledge. Besides I'm going to take a peek at zc.queue and compare it with my own implementation. Thanks and best regards, Manuel. Gary Poster wrote: On Feb 11, 2007, at 7:29 PM, Manuel Vazquez Acosta wrote: My scenario is akin a consumer-producer with shared buffer. Consumers pull items from the buffer whilst producers put items in the buffer. The buffer is an OOBTree along with an IOBTree which gives serial numbers to the keys of the OOBTree. I'm not sure if this is a match, but if you follow Chris' suggestions (which I recommend generally) then you might want to look at zc.queue: http://cheeseshop.python.org/pypi/zc.queue/ . Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Roché Compaan wrote: On Mon, 2008-02-25 at 07:36 +0200, Roché Compaan wrote: I'll update my blog post with the final stats and let you know when it is ready. I'll have to keep running these tests because the more I run them the faster the ZODB becomes ;-) Would you have guessed that the ZODB is faster at both insertion and lookups than Postgres? http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks-revisited Lookups are even faster then what I originally reported. Lookup times averages at 2 milliseconds (0.002s) on 10 million objects. I think somebody else should run these tests as well and validate the methodology behind them, otherwise I'm spreading lies. Roché, I'm very interested in your results. I'm assessing whether or not implement an application with ZODB. I have had a previous good experience, but was in a minor project. Now I need very fast lookups and retrievals, possibly distributed DB system and as easy as ZODB. Have done more tests recently? On the other hand I'm wondering too how this relate to Zope. Regards, Manuel. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Analyzing a ZODB.
In a development environment, set a breakpoint (e.g. add import pdb; pdb.set_trace() in ZODB.Connection.Connection.register. You'll be able to see exactly what is causing object changes. I recommend doing this in the zope debugger, which makes it easy to run one request at a time. Dear Mr. Fulton, Thanks for your quick response By the zope debugger you mean this: http://www.simplistix.co.uk/software/zope/zdb ?? Best regards, Manuel. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: Hooking persistent.Persistent.__setstate__ was Re: [ZODB-Dev] Analyzing a ZODB.
Alan Runyan wrote: - Customer has software on a remote machine. They are seeing unnecessary transaction commits. Just like the guy 'Analyzing a ZODB'. I'm that guy ;). BTW, we have related those unnecessary commits to CMFQuestions, an old plone product now superseded by PloneSurveys... We came to that not by inspecting the code, but by realizing there were too many conflicts related with CMFQuestionnaire. We removed it, and the commits vanished. Best regards, Manuel. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Zope memory usage
Alan, I'm replying to the Zope list also, because this issue is perhaps related to other components there. I'm running into the same situation: The python process running my Plone site is steadyly growing. I'm using Zope2.9.8-final (the one which works with Plone 2.5.5). What is the plan for Zope include such feature? Best regards, Manuel. Alan Runyan wrote: There was a recent modification to limit the ZODB cache to a set size. i.e. Limit the size of memory usage to 128MB. The original feature was implemented here: http://svn.zope.org/ZODB/branches/dm-memory_size_limited-cache/ You can get the feature+3.8 branch of the ZODB from: http://svn.zope.org/ZODB/branches/zcZODB-3.8/ The changes are also on trunk (will be ZODB 3.9). The goal of the modification is to try to put upper bounds on the size of the database cache. Before this size limited cache the cache would grow and if you had very large objects in the zodb cache -- your memory usage would be surpringsly high. Please use this feature in your testing at upfront. And let Roche know this feature has landed (if we was not monitoring svn checkins). The feature needs to be test driven. Give it a go and report your experience. cheers alan On Wed, Sep 17, 2008 at 5:10 AM, Izak Burger [EMAIL PROTECTED] wrote: Hi all, I'm sure this question has been asked before, but it drives me nuts so I figured I'll ask again. This is a problem that has been bugging me for ages. Why does zope memory use never decrease? Okay, I've seen it decrease maybe by a couple megabyte, but never by much. It seems the general way to run zope is to put in some kind of monitoring, and restart it when memory goes out of bounds. In general it always uses more and more RAM until the host starts paging to disk. This sort of baby-sitting just seems wrong to me. It doesn't seem to make any difference if you set the cache-size to a smaller number of objects or use a different number of threads. Over time things always go from good to bad and then on to worse. I have only two theories: a memory leak, or an issue with garbage collection (python side). It is possible that this is not a bug, my reasoning goes like this: The OS exposes a virtual memory picture to the application, ie, the application does not know how much of the RAM is real and how much is paged out. All it does is call malloc every now and then. Combined with a kernel that allows overcommit (which is in general a good thing), I see this going only one way. I'm hoping someone on this list has some insights into the issue. regards, Izak ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
