Re: [ZODB-Dev] Re: ZODB Benchmarks
Hi, On Fri, Mar 21, 2008 at 09:08:28PM +0100, Dieter Maurer wrote: Chris Withers wrote at 2008-3-20 22:22 +: Roché Compaan wrote: Not yet, they are very time consuming. I plan to do the same tests over ZEO next to determine what overhead ZEO introduces. Remember to try introducing more app servers and see where the bottleneck comes ;-) We have seen commit contention with lots (24) of zeo clients and a high write rate application (allmost all requests write to the ZODB). I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we should look into a technique called `group commit` to get rid of the commit contention. Does anybody know this technique already and maybe has a pointer for me? Christian -- gocept gmbh co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Summer of Code: students please sign up!
Hi there, If you're a student and you want to hack on the ZODB this summer in the Google Summer of Code, sign up soon: the deadline is just in a week's time! See here for more details: http://faassen.n--tree.net/blog/view/weblog/2008/03/25/0 Regards, Martijn ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Christian Theune wrote: I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we should look into a technique called `group commit` to get rid of the commit contention. Does anybody know this technique already and maybe has a pointer for me? I'd never heard the phrase until reading your message, but I think I got a pretty clear picture from http://forums.mysql.com/read.php?22,53854,53854#msg-53854 and http://archives.postgresql.org/pgsql-hackers/2007-03/msg01696.php. Summary: fsync is slow (and the cornerstone of most commit steps), so try to gather up a small batch of commits to do all at once (with only one call to fsync). Somewhat like Nagle's algorithm, but for fsync. The kicker is that OSs and hardware often lie about fsync (and it's therefore fast) and good hardware (disk arrays with battery backed write cache) already make fsync pretty fast. Not to suggest that group commit wouldn't speed things up, but it would seem that the technique will make the largest improvement for people that are using a non-lying fsync on inappropriate hardware. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] anecdotal speed comparison with RelStorage
Hey All, Just finished doing some rough speed tests having got RelStorage up and running against an Oracle 10g database using Oracle's InstantClient and cx_Oracle 4.3.3... The tests involved basically creating 10 folders, each with 10 pages in them, in a Plone site, using zope.testbrowser. FileStorage-over-ZEO managed the test in 3 mins 20 seconds Relatorage-to-Oracle managed the test in 3 mins 18 seconds So, not much in it, and shows that RelStorage is still a good thing (tm) even attached to Oracle :-) I'd be really interested to try some tests with multiple ZEO clients attached to a FileStorage versus multiple RelStorage client attached to Oracle. I suspect Oracle would win ;-) cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Benji York wrote at 2008-3-25 09:40 -0400: Christian Theune wrote: I talked to Brian Aker (MySQL guy) two weeks ago and he proposed that we should look into a technique called `group commit` to get rid of the commit contention. ... Summary: fsync is slow (and the cornerstone of most commit steps), so try to gather up a small batch of commits to do all at once (with only one call to fsync). Our commit contention definitely is not caused by fsync. Our fsync is quite fast. If only fsync would need to be considered, we could easily process at least 1.000 transactions per second -- but actually with 10 transactions per second we get contentions a few times per week. We do not yet precisely the cause of our commit contentions. Almost surely there are several causes that all can lead to contention. We already found: * client side causes (while the client helds to commit lock) - garbage collections (which can block a client in the order of 10 to 20 s) - NFS operations (which can take up to 27 s in our setup -- for still unknown reasons) - invalidation processing, espicially ZEO ClientCache processing * server side causes - commit lock hold during copy phase of pack - IO trashing during the reachability analysis in pack - non deterministic server side IO anomalities (IO suddently takes several times longer than usual -- for still unknown reasons) Somewhat like Nagle's algorithm, but for fsync. The kicker is that OSs and hardware often lie about fsync (and it's therefore fast) and good hardware (disk arrays with battery backed write cache) already make fsync pretty fast. Not to suggest that group commit wouldn't speed things up, but it would seem that the technique will make the largest improvement for people that are using a non-lying fsync on inappropriate hardware. -- Benji York Senior Software Engineer Zope Corporation -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Dieter Maurer wrote: We do not yet precisely the cause of our commit contentions. Hard to tell what'll make them better then. ;) Almost surely there are several causes that all can lead to contention. We already found: * client side causes (while the client helds to commit lock) - garbage collections (which can block a client in the order of 10 to 20 s) Interesting. Perhaps someone might enjoy investigating turning off garbage collection during commits. - NFS operations (which can take up to 27 s in our setup -- for still unknown reasons) Not much ZODB can do about that. ;) - invalidation processing, espicially ZEO ClientCache processing Interesting. Not knowing much about how invalidations are handled, I'm curious where the slow-down is. Do you have any more detail? * server side causes - commit lock hold during copy phase of pack - IO trashing during the reachability analysis in pack The new pack code should help quite a bit with the above (if you're saying what I think you're saying). - non deterministic server side IO anomalities (IO suddently takes several times longer than usual -- for still unknown reasons) Curious. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Benji York wrote at 2008-3-25 14:24 -0400: ... commit contentions ... Almost surely there are several causes that all can lead to contention. We already found: * client side causes (while the client helds to commit lock) - garbage collections (which can block a client in the order of 10 to 20 s) Interesting. Perhaps someone might enjoy investigating turning off garbage collection during commits. A reconfiguration of the garbage collector helped us with this one (the standard configuration is not well tuned to processes with large amounts of objects). - invalidation processing, espicially ZEO ClientCache processing Interesting. Not knowing much about how invalidations are handled, I'm curious where the slow-down is. Do you have any more detail? Not many: We have a component called RequestMonitor which periodically checks for long running requests and logs the corresponding stack traces. This monitor very often sees requests (holding the commit lock) which are in ZEO.cache.FileCache.settid. As the monitor runs asynchronously with the observed threads, the probability of an observation in a given function depends on how long the thread is inside this function (total time, i.e. visits times mean time per visit). From this, we can conclude that a significant time is spend in settid. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] anecdotal speed comparison with RelStorage
Chris Withers wrote: FileStorage-over-ZEO managed the test in 3 mins 20 seconds Relatorage-to-Oracle managed the test in 3 mins 18 seconds Cool, although it's not clear the storage speed had a major impact. If it did have a major impact, then the same test using plain FileStorage or RelStorage on MySQL should reveal that. MySQL was the clear leader in my own performance tests, despite my efforts to make PostgreSQL or Oracle win. ;) I've seen evidence that RelStorage on MySQL even has a chance of beating plain FileStorage in performance. I'd be really interested to try some tests with multiple ZEO clients attached to a FileStorage versus multiple RelStorage client attached to Oracle. I suspect Oracle would win ;-) I am certainly interested in hearing the results of that test. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Google Summer of Code 2008 - Student looking for Python Zope ZODB Grok Project / Mentor
Hi there: I would love to see if there is anyone looking for a student to mentor to do some hacking in fields such as Python Zope ZODB Grok etc., during this summer, in the Google Summer of Code 2008. If you are interested in learning more about my background and experiences please see my resume on : http://denali.cs.uh.edu:9080/mali/re/Mehmet_Ozbay_Resume-3-25-08.htm You can also view my profile on Google Summer of Code 2008 web space ( http://code.google.com/soc/2008/) in the applicant students list. Sincerely Mali Ozbay University Of Houston - Computer Science ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
