RE: [Zope] Max Connections
Pascal, Would you be so kind to send the zodb-part of your config? I Increased the zserver-threads for exactly the mentioned reasons but I was not aware of the connectio pool-parameter, nor can I find it anywhere in my zope.conf :) Thanks in Advance! /Carsten -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pascal Peregrina Sent: Friday, June 03, 2005 12:47 PM To: 'Nicholas Avenell'; zope@zope.org Subject: RE: [Zope] Max Connections 2 parameters in zope.conf : zserver-threads (main section) : the number of threads (i.e. incoming request) Zope will try to handle simultaneously But if you increase this value, you must also increase : pool-size (zodb_db/zodb_db section) : the connection pool size to the ZODB By default, zserver-threads is set to 4 and pool-size is set to 7 (as far as I could read in Zope code). So what I did is that I increased zserver-threads to 10 and pool-size to 15. It works fine for me. Pascal -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] la part de Nicholas Avenell Envoyé : vendredi 3 juin 2005 12:40 À : zope@zope.org Objet : [Zope] Max Connections Watching our Zope servers, they don't ever seem to be handling very many connections at a time. Where can I find out (and increase, probably) the Max number of connections Zope's serving? (This is Z2.6.4) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Max Connections
Yay, fine, thanks! And while we're at it: What exactly does cache-size 8000 do? 8000 what? Objects, Kilobytes? Per connection in Pool size or over all? Can someone point me out to a document explaining that? /Carsten -Original Message- From: Janko Hauser [mailto:[EMAIL PROTECTED] Sent: Friday, June 03, 2005 1:06 PM To: Germer, Carsten Cc: zope@zope.org Subject: Re: [Zope] Max Connections Am 03.06.2005 um 12:54 schrieb Germer, Carsten: Pascal, Would you be so kind to send the zodb-part of your config? I Increased the zserver-threads for exactly the mentioned reasons but I was not aware of the connectio pool-parameter, nor can I find it anywhere in my zope.conf :) Carsten look into lib/python/ZODB/component.xml There are all ZConfig directives defined and their defaults. From this I would assume the following ZODB config zodb_db main # Main FileStorage database filestorage path $INSTANCE/var/Data.fs /filestorage mount-point / cache-size 8000 pool-size 15 /zodb_db HTH, __Janko ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Max Connections
Thanks a bunch *off tweaking parameters* -Original Message- From: Pascal Peregrina [mailto:[EMAIL PROTECTED] Sent: Friday, June 03, 2005 1:45 PM To: Germer, Carsten Cc: 'zope@zope.org' Subject: RE: [Zope] Max Connections cache-size is the max number of objects each connection will try to keep in memory. You can see how it is currently performing when you go to the ZMI - Control Panel - Database Management - main - Cache Parameters Each line of the table is a connection of the pool. The total number of active an non active objects limited by the cache-size param. If you see, during normal operation, that most of the connections are reaching that limit, then you could try to increase it. But be careful not to run out of Memory ;) For ZEO users, there may be another parameter to change if you modify the cache-size param, which is the cache-size param within the zeoclient/zeoclient section. This param, although it has the same name, means something completely different. But this does not apply to the below example that uses a filestorage for the ZODB. Pascal -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] la part de Germer, Carsten Envoyé : vendredi 3 juin 2005 13:11 À : zope@zope.org Objet : RE: [Zope] Max Connections Yay, fine, thanks! And while we're at it: What exactly does cache-size 8000 do? 8000 what? Objects, Kilobytes? Per connection in Pool size or over all? Can someone point me out to a document explaining that? /Carsten -Original Message- From: Janko Hauser [mailto:[EMAIL PROTECTED] Sent: Friday, June 03, 2005 1:06 PM To: Germer, Carsten Cc: zope@zope.org Subject: Re: [Zope] Max Connections Am 03.06.2005 um 12:54 schrieb Germer, Carsten: Pascal, Would you be so kind to send the zodb-part of your config? I Increased the zserver-threads for exactly the mentioned reasons but I was not aware of the connectio pool-parameter, nor can I find it anywhere in my zope.conf :) Carsten look into lib/python/ZODB/component.xml There are all ZConfig directives defined and their defaults. From this I would assume the following ZODB config zodb_db main # Main FileStorage database filestorage path $INSTANCE/var/Data.fs /filestorage mount-point / cache-size 8000 pool-size 15 /zodb_db HTH, __Janko ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com ** ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Tweaking Zope DB- and connection parameters
Hello everyone, I'm trying to figure out what I can set all (I know) Zope Parameters to to get a good overall performance on our old live installation. We have a new System with upgraded Zope and the like in the making but untill then I'm trying to keep what is up smooth and stable. Machine: Dual Pentium with 4Gig Ram running on Suse Linux. Zope 2.7.2 with Python 2.3.4 and ZMS for the Sites. There's a recent Apache 1.3.x in Front for caching and like 25 Sites running on the system, some very rarely visited some more loaded. While running the system with out-of-the-box-parameters at times of high load the RAM went full, the machine started swapping and spiraling down so I started to set limits in Apache and Zope. I was mostly just guessing! Apache is set to MaxClients 50 Zope.conf has set zserver-threads 5 and zodb_db main cache-size 500 pool-size 25 ... /zodb_db With this settings the machine tops at load around 1.8 - 2 and no swapping takes place *puuh* but still users sometimes, too often for users taste, get to wait a minute and longer for their page. For me it tastes like a garbage collection or whatnot because after 5-20 minutes everything is fine again for an hour or two... - Apaches MaxClients is set for general limit - zserver-threads: How many ppl, approx., can connect with this set to 5? Would be good to synchronize this with the apache settings somehow? - cache-size: This is per DB connection (pool-size) as I have learned, I did set this down to limit memory usage - pool-size: number of concurrent DB-Connections as I learned. Does every concurrent connection have to have it's own db-connection or do N zserver-threads share one? What I'm looking for is general advice how these parameters work and can be set for best cooperation. Or if there are other Params that I have missed yet ;) E.g. every zserver-thread uses one db-connection and can server one user so for 50 concurrent users you have to set everything to 50 and tweak cache-size according to 'Cache Parameters' in the Zope config and your RAM-load behaves Which is nonsense I bet but something like that would help me great time. Thanks in Advance for any Help! /Carsten Carsten Germer Deutsches Elektronen Synchrotron (Web-Office, IT) phone: +49-40-8998-1661Notkestr. 85 web: http://wof.desy.de22607 Hamburg e-mail: [EMAIL PROTECTED] Germany ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Tweaking Zope DB- and connection parameters
Ahh, this clarifies so far, thanks. But about how many zserver-threads would one need for approx. 50 concurrent connections? /Carsten -Original Message- From: Pascal Peregrina [mailto:[EMAIL PROTECTED] Sent: Tuesday, June 28, 2005 11:54 AM To: Germer, Carsten; zope list user Subject: RE: [Zope] Tweaking Zope DB- and connection parameters zserver-threads and pool-size should be almost the same. Each HTTP request from Apache to Zope uses a Connection object bound to a Transaction object. As each Connection object has its own cache, then if the connection pool size is much higher than the number of zsever threads, you could end up using new connections with empty cache when it's not necessary... This will make the request response time slower (cause the connection will need to get all objects from ZODB) and will end up using unecessary memory (cause all connection caches will be caching objects). I would recommend pool-size 8 if you keep zserver-threads to 5. (I don't know why pool-size must be a little higher than threads, but the default configuration is 4 threads and 7 connections, so...) Then try increasing the cache size, because if you have lots of different websites, caching 500 objects will probably cause each connection to often load objects from ZODB (cache miss). To monitor that you can use Control Panel - Database Management - Main - Cache Parameters For each connection, if the number of active objects hits your 500 limit, then it may be a good idea to increase it. But of course this will depend oh the available memory you have. Hope this helps. Pascal -Message d'origine- De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] la part de Germer, Carsten Envoyé : mardi 28 juin 2005 11:32 À : zope list user Objet : [Zope] Tweaking Zope DB- and connection parameters Hello everyone, I'm trying to figure out what I can set all (I know) Zope Parameters to to get a good overall performance on our old live installation. We have a new System with upgraded Zope and the like in the making but untill then I'm trying to keep what is up smooth and stable. Machine: Dual Pentium with 4Gig Ram running on Suse Linux. Zope 2.7.2 with Python 2.3.4 and ZMS for the Sites. There's a recent Apache 1.3.x in Front for caching and like 25 Sites running on the system, some very rarely visited some more loaded. While running the system with out-of-the-box-parameters at times of high load the RAM went full, the machine started swapping and spiraling down so I started to set limits in Apache and Zope. I was mostly just guessing! Apache is set to MaxClients 50 Zope.conf has set zserver-threads 5 and zodb_db main cache-size 500 pool-size 25 ... /zodb_db With this settings the machine tops at load around 1.8 - 2 and no swapping takes place *puuh* but still users sometimes, too often for users taste, get to wait a minute and longer for their page. For me it tastes like a garbage collection or whatnot because after 5-20 minutes everything is fine again for an hour or two... - Apaches MaxClients is set for general limit - zserver-threads: How many ppl, approx., can connect with this set to 5? Would be good to synchronize this with the apache settings somehow? - cache-size: This is per DB connection (pool-size) as I have learned, I did set this down to limit memory usage - pool-size: number of concurrent DB-Connections as I learned. Does every concurrent connection have to have it's own db-connection or do N zserver-threads share one? What I'm looking for is general advice how these parameters work and can be set for best cooperation. Or if there are other Params that I have missed yet ;) E.g. every zserver-thread uses one db-connection and can server one user so for 50 concurrent users you have to set everything to 50 and tweak cache-size according to 'Cache Parameters' in the Zope config and your RAM-load behaves Which is nonsense I bet but something like that would help me great time. Thanks in Advance for any Help! /Carsten --- - Carsten Germer Deutsches Elektronen Synchrotron (Web-Office, IT) phone: +49-40-8998-1661 Notkestr. 85 web: http://wof.desy.de 22607 Hamburg e-mail: [EMAIL PROTECTED] Germany --- - ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received
[Zope-dev] AW: Caching prob with AHCM and headers
Well, sadly I didn't find the time to pursue the issue any further yet. We're kneedeep in work to get the system up and running and caching isn't top priority so far. I don't even know if someone patched the AHCM yet, simply haven't looked. But as someone suggested somewhere and you can see in http://mail.zope.org/pipermail/zope/2003-April/135101.html it works if you set the headers yourself. So if you need caching _now_ and AHCM is not fixed yet, change the object used so that it sets the headers correctly itself. I know that this solution is ikky but I don't know when I will have time to tackle the issue of a comfortable cache system. Though I know I will have to solve it sometime :) /Carsten -Ursprüngliche Nachricht- Von: Bjorn Stabell [mailto:[EMAIL PROTECTED] Gesendet: Mittwoch, 10. September 2003 13:07 An: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED] Betreff: Re: Caching prob with AHCM and headers Accelerated HTTP Caching Manager doesn't work out-of-the-box as this thread reported, but the thread had no conclusion: http://mail.zope.org/pipermail/zope/2003-April/134800.html http://mail.zope.org/pipermail/zope/2003-April/135059.html http://mail.zope.org/pipermail/zope/2003-April/135101.html I'm running into the same problem now, also with Apache as the caching reverse proxy, so I was wondering if there was any progress? Here are some of my thoughts; It seems reasonable that without a Last-Modified or Etag header in the response, that a client (and surrogate/ http accelerator) will not have any validators (e.g., If-Modified-Since or If-Match corresponding to last modified and etag headers, respectively) in the request headers. It's probably right that dynamic code doesn't return a Last-Modified header; instead, the RFC includes an Etag header, which is hash of the content, which is much easier to use for dynamic code. The Etag header returned by Zope, however, looks very suspicous; it is empty. This could perhaps be fooling the caching machinery? Perhaps the Etag header should just always be a hash of the content? (Another system worth looking at: http://www.jpcache.com/) Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] AW: [Summary] Caching prob with AHCM and headers
Thanks Bjorn, now I remember the last-modified issue. As I remember it, someone said that the line is commented out for a reason not known... that's why I didn't change it :) /Carsten -Ursprüngliche Nachricht- Von: Bjorn Stabell [mailto:[EMAIL PROTECTED] Gesendet: Freitag, 12. September 2003 12:14 An: [EMAIL PROTECTED] Cc: Janko Hauser; [EMAIL PROTECTED]; [EMAIL PROTECTED]; Germer, Carsten Betreff: [Summary] Caching prob with AHCM and headers Just for the record to help others in the same situation... PROBLEM Apache (1.3.28 and 2.0.47) will not cache content with just an Expires header, but it will correctly cache content with a Last-Modified header. This problem has been reported before in 1999 by James Cooper, but the bug request seems to have been forgotten/ left behind. It makes sense for dynamic content to produce Expires headers, not Last-Modified headers, so this is a problem. Looking at the code for Zope, it seems that the standard HTTP accelerator product used to produce a Last-Modified header, but no longer does so, thus making it useless with the current versions of Apache. The problem has been re-reported to the Apache group with patches: http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23129 (for Apache 1.3.28) http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23130 (for Apache 2.0.47) WORK-AROUNDS 1) Produce a Last-Modified header yourself by using 2) Patch Zope's standard HTTP accelerator product so that it produces a Last-Modified header (the line is commented out now) SOLUTIONS 3) Patch Apache (see bug report links above for patches) 4) Wait until the problem is hopefully fixed in Apache Bye, -- Bjorn Stabell [EMAIL PROTECTED] Tel +86 (10) 65918490 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-DB] Closing idle DCoracle2 Connections?
Ok, that's a bit much for my question I guess... I gather from all your mails that I can not close a database connection periodically or on event from Zope (Script or whatever)? Any hands-down clues what I can do to prevent db-conns from ideling around? /Carsten -Original Message- From: Dieter Maurer [mailto:[EMAIL PROTECTED] Sent: Saturday, October 22, 2005 8:07 PM To: Matthew T. Kromer Cc: Chris Withers; Germer, Carsten; zope-db@zope.org Subject: Re: [Zope-DB] Closing idle DCoracle2 Connections? Matthew T. Kromer wrote at 2005-10-21 20:39 -0400: ... What happens with Zope and DCOracle2 I think is when the DA connection object is ghosted, it doesn't know about it, and thus doesn't close its connection. Why not? When the DA instance is ghosted, the db object is deleted (last reference released). In an optimal world, the deletion of the db object would release the last reference to the DCOracle connection and this would close the connection to Oracle before the memory is released. ... The close is implicit when the connection object is destroyed, but usually there's so much stuff loaded that *someone* has a reference to the connection. That's the bug. There should be no references to DCOracle resources outside the db object (at least, Zope's DA/ZSQL do not require such references outside). Thus, when the db object goes away, the DCOracle resources should, too. However, the code is still geared around OCI8 and some Oracle 9 code, its due for some refactoring in a big way to deal with the profusion of handle-driven objects now much more prevalent in Oracle 10. -- Dieter ___ Zope-DB mailing list Zope-DB@zope.org http://mail.zope.org/mailman/listinfo/zope-db
