Re: [Zope] Zope scalabilty and problems
On 9/1/05, Kennamore, Matthew G [NTK] <[EMAIL PROTECTED]> wrote: > Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) > Pyhton is 2.3.4 > Apache 1.3 > > We have 943 users as of this minute with a bout 1000 objects being created a > day (Lots of creates) Interesting. Somebody have a couple of values for heavy load sites (Zope only and with Plone) ? - hit per hour & day - visitor per hour & day - Gb of the ZODB Regards. -- Sébastien Douche <[EMAIL PROTECTED]> ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope scalabilty and problems
Andrew Sawyers wrote: Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1, writes a day is not outrageous Yes, but he's using Plone, which implies not only all of the CMF reindexing overhead, but also all the AT and Plone layers on top. That could get even worse if he's gone to town with a workflow tool or is using any of the truly appalling Plone addons, such as CMFMember, CMFForum, etc... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Zope scalabilty and problems
Here is some of the information I can provide, some I'll have to go check. Database: In the past hour we had 1204 Object Stores and 159433 Loads and 465 connections. -Original Message- From: Andrew Sawyers [mailto:[EMAIL PROTECTED] Sent: Thursday, September 01, 2005 1:30 PM To: Kennamore, Matthew G [NTK]; 'Chris McDonough' Cc: zope@zope.org Subject: RE: [Zope] Zope scalabilty and problems Sorry, I missed the beginning of this thread, I've got some questions that I didn't see answered or asked: Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1, writes a day is not outrageousso it's possible that really what you need to spread the load across a load balanced pool of zeo app servers. How many users are authenticated at any given time actually using the system for writing? Are you running a persistent or non-persistent cache? If so, what size it is? Check the control panel, under database management and choose your database, then look at the activity tab. Do you have a high volume of object loads? What is the volume of object stores? There is also an in-memory zeo object cache - click the cache parameters tab and see the number of objects in your in-memory cache. If your zeo client(s) have lots of ram (they definitely should), you can crank this up as well. What is the 'horespower' of your app server(s)? Andrew -- Zope Managed Hosting Software Engineer Zope Corporation (540) 361-1700 > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of > Kennamore, Matthew G [NTK] > Sent: Thursday, September 01, 2005 1:19 PM > To: Chris McDonough > Cc: zope@zope.org > Subject: RE: [Zope] Zope scalabilty and problems > > Thanks everyone for their suggestions, bought a squid book today and I > know what I'll be doing all weekend > > -Original Message- > From: Chris McDonough [mailto:[EMAIL PROTECTED] > Sent: Thursday, September 01, 2005 1:13 PM > To: Kennamore, Matthew G [NTK] > Cc: zope@zope.org > Subject: RE: [Zope] Zope scalabilty and problems > > What you've got now is a pretty reasonable setup. Moving to 2.7.7 will > likely not get you any "free" performance increases and it doesn't > appear that there are any "critical" ZODB bugfixes relating to your > problems since 2.7.3. FWIW, also, I'd venture a guess that moving "off > Data.fs" (as you mentioned in your original email) may be quite costly, > requiring a lot of application recoding. It also might not help solve > the scaling problem you're having. It's a "high risk, potentially low > reward" solution. You need something that is low-risk and potentially > high reward, I'd suspect. > > Others have said this, but it bears repeating. The biggest > "administrative" win you can get for the cheapest cost is to add an HTTP > cache (like Squid or even Apache's mod_proxy) in front of your ZEO > clients and to cause your Zope application to set the proper response > headers which allow the cache server to cache (and thus serve) those > pages on subsequent requests. > > This can be tricky when most of your traffic is authenticated because > there is usually very little win and a lot of danger to serving up > cached pages when a page in the cache represents a view that a > privileged user sees when visiting the site. You typically don't want > other people to see the same thing he does. As a result, typically > people will say "pages that require authentication can't be cached" > which isn't entirely true (you can cache them but it's often useless). > > However, even serving up fully static content that doesn't require any > authentication like CSS, images, and so on can be a win. You might > start there. > > HTH, > > - C > > > On Thu, 2005-09-01 at 11:54 -0500, Kennamore, Matthew G [NTK] wrote: > > Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) > > Pyhton is 2.3.4 > > Apache 1.3 > > > > We have 943 users as of this minute with a bout 1000 objects being > created a day (Lots of creates) > > > > -Original Message- > > From: Chris McDonough [mailto:[EMAIL PROTECTED] > > Sent: Thursday, September 01, 2005 12:51 PM > > To: Kennamore, Matthew G [NTK] > > Cc: zope@zope.org > > Subject: Re: [Zope] Zope scalabilty and problems > > > > Your "tranaaction blocked" error messages seem to imply that either or > > both of the following is true: > > > > - some transactions are taking a "long time" > > (any more t
RE: [Zope] Zope scalabilty and problems
Sorry, I missed the beginning of this thread, I've got some questions that I didn't see answered or asked: Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1, writes a day is not outrageousso it's possible that really what you need to spread the load across a load balanced pool of zeo app servers. How many users are authenticated at any given time actually using the system for writing? Are you running a persistent or non-persistent cache? If so, what size it is? Check the control panel, under database management and choose your database, then look at the activity tab. Do you have a high volume of object loads? What is the volume of object stores? There is also an in-memory zeo object cache - click the cache parameters tab and see the number of objects in your in-memory cache. If your zeo client(s) have lots of ram (they definitely should), you can crank this up as well. What is the 'horespower' of your app server(s)? Andrew -- Zope Managed Hosting Software Engineer Zope Corporation (540) 361-1700 > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of > Kennamore, Matthew G [NTK] > Sent: Thursday, September 01, 2005 1:19 PM > To: Chris McDonough > Cc: zope@zope.org > Subject: RE: [Zope] Zope scalabilty and problems > > Thanks everyone for their suggestions, bought a squid book today and I > know what I'll be doing all weekend > > -Original Message- > From: Chris McDonough [mailto:[EMAIL PROTECTED] > Sent: Thursday, September 01, 2005 1:13 PM > To: Kennamore, Matthew G [NTK] > Cc: zope@zope.org > Subject: RE: [Zope] Zope scalabilty and problems > > What you've got now is a pretty reasonable setup. Moving to 2.7.7 will > likely not get you any "free" performance increases and it doesn't > appear that there are any "critical" ZODB bugfixes relating to your > problems since 2.7.3. FWIW, also, I'd venture a guess that moving "off > Data.fs" (as you mentioned in your original email) may be quite costly, > requiring a lot of application recoding. It also might not help solve > the scaling problem you're having. It's a "high risk, potentially low > reward" solution. You need something that is low-risk and potentially > high reward, I'd suspect. > > Others have said this, but it bears repeating. The biggest > "administrative" win you can get for the cheapest cost is to add an HTTP > cache (like Squid or even Apache's mod_proxy) in front of your ZEO > clients and to cause your Zope application to set the proper response > headers which allow the cache server to cache (and thus serve) those > pages on subsequent requests. > > This can be tricky when most of your traffic is authenticated because > there is usually very little win and a lot of danger to serving up > cached pages when a page in the cache represents a view that a > privileged user sees when visiting the site. You typically don't want > other people to see the same thing he does. As a result, typically > people will say "pages that require authentication can't be cached" > which isn't entirely true (you can cache them but it's often useless). > > However, even serving up fully static content that doesn't require any > authentication like CSS, images, and so on can be a win. You might > start there. > > HTH, > > - C > > > On Thu, 2005-09-01 at 11:54 -0500, Kennamore, Matthew G [NTK] wrote: > > Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) > > Pyhton is 2.3.4 > > Apache 1.3 > > > > We have 943 users as of this minute with a bout 1000 objects being > created a day (Lots of creates) > > > > -Original Message- > > From: Chris McDonough [mailto:[EMAIL PROTECTED] > > Sent: Thursday, September 01, 2005 12:51 PM > > To: Kennamore, Matthew G [NTK] > > Cc: zope@zope.org > > Subject: Re: [Zope] Zope scalabilty and problems > > > > Your "tranaaction blocked" error messages seem to imply that either or > > both of the following is true: > > > > - some transactions are taking a "long time" > > (any more than, say, 200 milliseconds is a long time) > > > > - you have a very high transaction volume. > > > > That said, these messages are informational rather than signifying an > > error condition. Some transaction blockage is expected since the ZEO > > server can only deal with one transaction at a time. > > > > What version of Zope are you using? > > > > Is this an "intranet"
RE: [Zope] Zope scalabilty and problems
Thanks everyone for their suggestions, bought a squid book today and I know what I'll be doing all weekend -Original Message- From: Chris McDonough [mailto:[EMAIL PROTECTED] Sent: Thursday, September 01, 2005 1:13 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject: RE: [Zope] Zope scalabilty and problems What you've got now is a pretty reasonable setup. Moving to 2.7.7 will likely not get you any "free" performance increases and it doesn't appear that there are any "critical" ZODB bugfixes relating to your problems since 2.7.3. FWIW, also, I'd venture a guess that moving "off Data.fs" (as you mentioned in your original email) may be quite costly, requiring a lot of application recoding. It also might not help solve the scaling problem you're having. It's a "high risk, potentially low reward" solution. You need something that is low-risk and potentially high reward, I'd suspect. Others have said this, but it bears repeating. The biggest "administrative" win you can get for the cheapest cost is to add an HTTP cache (like Squid or even Apache's mod_proxy) in front of your ZEO clients and to cause your Zope application to set the proper response headers which allow the cache server to cache (and thus serve) those pages on subsequent requests. This can be tricky when most of your traffic is authenticated because there is usually very little win and a lot of danger to serving up cached pages when a page in the cache represents a view that a privileged user sees when visiting the site. You typically don't want other people to see the same thing he does. As a result, typically people will say "pages that require authentication can't be cached" which isn't entirely true (you can cache them but it's often useless). However, even serving up fully static content that doesn't require any authentication like CSS, images, and so on can be a win. You might start there. HTH, - C On Thu, 2005-09-01 at 11:54 -0500, Kennamore, Matthew G [NTK] wrote: > Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) > Pyhton is 2.3.4 > Apache 1.3 > > We have 943 users as of this minute with a bout 1000 objects being created a > day (Lots of creates) > > -Original Message- > From: Chris McDonough [mailto:[EMAIL PROTECTED] > Sent: Thursday, September 01, 2005 12:51 PM > To: Kennamore, Matthew G [NTK] > Cc: zope@zope.org > Subject: Re: [Zope] Zope scalabilty and problems > > Your "tranaaction blocked" error messages seem to imply that either or > both of the following is true: > > - some transactions are taking a "long time" > (any more than, say, 200 milliseconds is a long time) > > - you have a very high transaction volume. > > That said, these messages are informational rather than signifying an > error condition. Some transaction blockage is expected since the ZEO > server can only deal with one transaction at a time. > > What version of Zope are you using? > > Is this an "intranet" application? Are there lots of users creating > content? Can you venture a guess as to how many users are "creating" > content vs. "viewing" content at any given time? Are the users that are > creating content logged in? The users that are viewing content? > > On Wed, 2005-08-31 at 11:03 -0500, Kennamore, Matthew G [NTK] wrote: > > Here is our situation, we originally launched zope with a small user > > base in mind and like most IT implementations it got away from us > > quickly and we have over 1000 users now. We are experiencing huge > > slowdowns and lots of complaints of performance. > > > > Let me outline our system: > > > > I have a load balancer that round robins incoming request to 3 zeo > > servers > > > > I have 1 zope server on the back en that severs the data, this is a > > large single processor machine with 4 gigs of ram > > > > Each zeo server is 2 gigs of ram, single processor 3.2 gighz > > > > The throughput doesn't appear to be the problem as ram/processors are. > > > > Short term I plan to change all zeo's to 4 gigs ram and add 3 > > additional servers. > > > > > > The Technologies we are using are Zope, Zeo, plone and so forth. Why > > are we having these large issues? I see sites with hundreds of > > thousands of users and they don't seem to have these issues? We plan > > to recode our site to get away from the Data.fs as soon as we can but > > that is a ways off yet. What am I doing wrong can anyone help? > > > > ___ > > Zope mailli
RE: [Zope] Zope scalabilty and problems
What you've got now is a pretty reasonable setup. Moving to 2.7.7 will likely not get you any "free" performance increases and it doesn't appear that there are any "critical" ZODB bugfixes relating to your problems since 2.7.3. FWIW, also, I'd venture a guess that moving "off Data.fs" (as you mentioned in your original email) may be quite costly, requiring a lot of application recoding. It also might not help solve the scaling problem you're having. It's a "high risk, potentially low reward" solution. You need something that is low-risk and potentially high reward, I'd suspect. Others have said this, but it bears repeating. The biggest "administrative" win you can get for the cheapest cost is to add an HTTP cache (like Squid or even Apache's mod_proxy) in front of your ZEO clients and to cause your Zope application to set the proper response headers which allow the cache server to cache (and thus serve) those pages on subsequent requests. This can be tricky when most of your traffic is authenticated because there is usually very little win and a lot of danger to serving up cached pages when a page in the cache represents a view that a privileged user sees when visiting the site. You typically don't want other people to see the same thing he does. As a result, typically people will say "pages that require authentication can't be cached" which isn't entirely true (you can cache them but it's often useless). However, even serving up fully static content that doesn't require any authentication like CSS, images, and so on can be a win. You might start there. HTH, - C On Thu, 2005-09-01 at 11:54 -0500, Kennamore, Matthew G [NTK] wrote: > Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) > Pyhton is 2.3.4 > Apache 1.3 > > We have 943 users as of this minute with a bout 1000 objects being created a > day (Lots of creates) > > -Original Message- > From: Chris McDonough [mailto:[EMAIL PROTECTED] > Sent: Thursday, September 01, 2005 12:51 PM > To: Kennamore, Matthew G [NTK] > Cc: zope@zope.org > Subject: Re: [Zope] Zope scalabilty and problems > > Your "tranaaction blocked" error messages seem to imply that either or > both of the following is true: > > - some transactions are taking a "long time" > (any more than, say, 200 milliseconds is a long time) > > - you have a very high transaction volume. > > That said, these messages are informational rather than signifying an > error condition. Some transaction blockage is expected since the ZEO > server can only deal with one transaction at a time. > > What version of Zope are you using? > > Is this an "intranet" application? Are there lots of users creating > content? Can you venture a guess as to how many users are "creating" > content vs. "viewing" content at any given time? Are the users that are > creating content logged in? The users that are viewing content? > > On Wed, 2005-08-31 at 11:03 -0500, Kennamore, Matthew G [NTK] wrote: > > Here is our situation, we originally launched zope with a small user > > base in mind and like most IT implementations it got away from us > > quickly and we have over 1000 users now. We are experiencing huge > > slowdowns and lots of complaints of performance. > > > > Let me outline our system: > > > > I have a load balancer that round robins incoming request to 3 zeo > > servers > > > > I have 1 zope server on the back en that severs the data, this is a > > large single processor machine with 4 gigs of ram > > > > Each zeo server is 2 gigs of ram, single processor 3.2 gighz > > > > The throughput doesn't appear to be the problem as ram/processors are. > > > > Short term I plan to change all zeo's to 4 gigs ram and add 3 > > additional servers. > > > > > > The Technologies we are using are Zope, Zeo, plone and so forth. Why > > are we having these large issues? I see sites with hundreds of > > thousands of users and they don't seem to have these issues? We plan > > to recode our site to get away from the Data.fs as soon as we can but > > that is a ways off yet. What am I doing wrong can anyone help? > > > > ___ > > Zope maillist - Zope@zope.org > > http://mail.zope.org/mailman/listinfo/zope > > ** No cross posts or HTML encoding! ** > > (Related lists - > > http://mail.zope.org/mailman/listinfo/zope-announce > > http://mail.zope.org/mailman/listinfo/zope-dev ) > > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Zope scalabilty and problems
Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) Pyhton is 2.3.4 Apache 1.3 We have 943 users as of this minute with a bout 1000 objects being created a day (Lots of creates) -Original Message- From: Chris McDonough [mailto:[EMAIL PROTECTED] Sent: Thursday, September 01, 2005 12:51 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject: Re: [Zope] Zope scalabilty and problems Your "tranaaction blocked" error messages seem to imply that either or both of the following is true: - some transactions are taking a "long time" (any more than, say, 200 milliseconds is a long time) - you have a very high transaction volume. That said, these messages are informational rather than signifying an error condition. Some transaction blockage is expected since the ZEO server can only deal with one transaction at a time. What version of Zope are you using? Is this an "intranet" application? Are there lots of users creating content? Can you venture a guess as to how many users are "creating" content vs. "viewing" content at any given time? Are the users that are creating content logged in? The users that are viewing content? On Wed, 2005-08-31 at 11:03 -0500, Kennamore, Matthew G [NTK] wrote: > Here is our situation, we originally launched zope with a small user > base in mind and like most IT implementations it got away from us > quickly and we have over 1000 users now. We are experiencing huge > slowdowns and lots of complaints of performance. > > Let me outline our system: > > I have a load balancer that round robins incoming request to 3 zeo > servers > > I have 1 zope server on the back en that severs the data, this is a > large single processor machine with 4 gigs of ram > > Each zeo server is 2 gigs of ram, single processor 3.2 gighz > > The throughput doesn't appear to be the problem as ram/processors are. > > Short term I plan to change all zeo's to 4 gigs ram and add 3 > additional servers. > > > The Technologies we are using are Zope, Zeo, plone and so forth. Why > are we having these large issues? I see sites with hundreds of > thousands of users and they don't seem to have these issues? We plan > to recode our site to get away from the Data.fs as soon as we can but > that is a ways off yet. What am I doing wrong can anyone help? > > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope scalabilty and problems
Your "tranaaction blocked" error messages seem to imply that either or both of the following is true: - some transactions are taking a "long time" (any more than, say, 200 milliseconds is a long time) - you have a very high transaction volume. That said, these messages are informational rather than signifying an error condition. Some transaction blockage is expected since the ZEO server can only deal with one transaction at a time. What version of Zope are you using? Is this an "intranet" application? Are there lots of users creating content? Can you venture a guess as to how many users are "creating" content vs. "viewing" content at any given time? Are the users that are creating content logged in? The users that are viewing content? On Wed, 2005-08-31 at 11:03 -0500, Kennamore, Matthew G [NTK] wrote: > Here is our situation, we originally launched zope with a small user > base in mind and like most IT implementations it got away from us > quickly and we have over 1000 users now. We are experiencing huge > slowdowns and lots of complaints of performance. > > Let me outline our system: > > I have a load balancer that round robins incoming request to 3 zeo > servers > > I have 1 zope server on the back en that severs the data, this is a > large single processor machine with 4 gigs of ram > > Each zeo server is 2 gigs of ram, single processor 3.2 gighz > > The throughput doesn’t appear to be the problem as ram/processors are. > > Short term I plan to change all zeo’s to 4 gigs ram and add 3 > additional servers. > > > The Technologies we are using are Zope, Zeo, plone and so forth. Why > are we having these large issues? I see sites with hundreds of > thousands of users and they don’t seem to have these issues? We plan > to recode our site to get away from the Data.fs as soon as we can but > that is a ways off yet. What am I doing wrong can anyone help? > > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Zope scalabilty and problems
I am seeing a large amounts of these 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.12:50602 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.23:49161 Blocked transaction restarted. 2005-08-31T09:05:21 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T09:05:22 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:18:26 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:18:28 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:08 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:59:17 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:31 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:13 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T11:00:15 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:16 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 2. -Original Message- From: Paul Winkler [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 31, 2005 1:33 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject:Re: [Zope] Zope scalabilty and problems > Here is our situation, we originally launched zope with a small user > base in mind and like most IT implementations it got away from us > quickly and we have over 1000 users now. We are experiencing huge > slowdowns and lots of complaints of performance. > > Let me outline our system: > > I have a load balancer that round robins incoming request to 3 zeo > servers Just to get the terminology correct: These are Zope servers. They are ZEO *clients*. > I have 1 zope server on the back en that severs the data, this is a > large single processor machine with 4 gigs of ram I think you mean ZEO server, not Zope server. > Each zeo server is 2 gigs of ram, single processor 3.2 gighz That is plenty of hardware for most sites... > Short term I plan to change all zeo's to 4 gigs ram and add 3 additional > servers. That might help if your users' typical usage patterns are unavoidably dynamic: e.g. lots of unique catalog queries, lots of personalization, and/or lots of writes. But it's more typical to have quite a lot of reads of rarely-changing data. In which case, the usual low-cost answer to performance issues is: cache, cache, and more cache. There are several levels of caching, which vary in difficulty of setup and potential performance impact. See below... > The Technologies we are using are Zope, Zeo, plone and so forth. Why > are we having these large issues? I see sites with hundreds of > thousands of users and they don't seem to have these issues? We plan to > recode our site to get away from the Data.fs as soon as we can but that > is a ways off yet. What am I doing wrong can anyone help? Plone is very highly configurable, and every one of those runtime options adds a performance cost. You certainly can run high-performance plone sites on your hardware, you just need to tune it properly. The general steps are: 1) Test your performance. See http://plope.com/Members/chrism/loadtesting-methodology/view?searchterm=performance 2) Profile to see what's slow. 3) Increase caching and/or optimize slow code identified by profiling. Caching is generally a bigger win than optimizing, for less effort. (I have been doing full-time zope work for about 4 years now and I have yet to ever consciously optimize any code ... I try to have efficiency in mind while writing it but that's it... so far there's always been a simple solution via caching. But I don't work on mega-heavy-load sites.) 4) Repeat until satisfied. The best docs I've found on this are: "Scaling Zope", Chris McDonough's presentation from 2003: http://plope.com/misc/szweb Read that one from start to finish at least once. There is also a good chapter on maintaining & optimizing plone in Andy McKay's "Plone book". Another good one is http://web.archive.org/web/20030723200058/http://matthardy.us/mywiki/HighPerformanceZope but the original has gone missing so the charts aren't there. -PW ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope scalabilty and problems
Am Mittwoch, 31. August 2005 18:50 schrieb J Cameron Cooper: > There are lots of ways to improve performance just administratively. You > can improve various zope.conf settings, and set up caching (either RAM > or HTTP Accel.) in Plone. You can also put a caching proxy server (like > Squid) in front of your Zopes. not to forget that playing with the zeo and/or zope database caches might help; if you see lots of conflict errors in you logs, may be an upgrade to 2.8 with MVCC does a benefit. and, of course, look carefully if any other tasks on all of your servers compete with your cpu and io resources. Cheers, Sascha P.S. just in case, debug mode or profiling and such are better to be disabled on your live system...:-) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope scalabilty and problems
Kennamore, Matthew G [NTK] wrote: The Technologies we are using are Zope, Zeo, plone and so forth. Why are we having these large issues? It depends entirely on your application. Plone doesn't help you here, and I'd certainly consider replacing that long before I think about trying to ditch ZODB in favour of a relation back end (something I find confusing, since a relational db can be just as slow if you mistreat it in the ways people often mistreat ZODB - "Data.fs", ie: FileStorage, is about as fast as it gets when used correctly. As I said, Plone doesn't help you here, and it, and the coding styles it encourages, along with Archetypes myriad layers of indirection are likely to be the source of your problem. I'd start with: - make sure requests that shouldn't be writing to the ZODB aren't writing to the ZODB - The Undo tab in the root of your Zope instance is good for that, it shouldn't have any entries for views on it - get PTProfiler up and running and profile your most commonly used urls, try and knock on the top file slowest items on the profile results, or at least make them not slow anymore - look at doing some clever coding, portal_catalog search and AT references are the worst offendors for hopelessly slow code that you'd expect to be fast. - finally, start looking at app-level caching, things like CMF's cookedBody can be applied more widely as techniques. - at this point, look at systemic caching such as Squid, ESI, etc. This is much harder if you need to have authenticated users. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope scalabilty and problems
>From experience, one of the most efficient performance boosts is to purchase a fast machine for it. If you're in a commercial environment, your shareholders might prefer that you spent your costed time developing new things instead of spending a few extra hundreds of pounds/dollars on expensive CPUs. We even have a plone 2.0 site that is quite fast believe it or not. And yes, as J says, get the worst bottlenecks out of the way, but bare in mind that it's only a bottleneck if it's requested often (such as the homepage). If you are relying on the Data.fs for complex data handling, make sure that your common read methods don't make any writes to the zodb. I had such a "bug" in the issuetrackerproduct where getIssueFiltered() made a save to the zodb under certain conditions. Under heavy use I started to get zodb conflict errors that I didn't see when developing on localhost. On 8/31/05, J Cameron Cooper <[EMAIL PROTECTED]> wrote: > Kennamore, Matthew G [NTK] wrote: > > Here is our situation, we originally launched zope with a small user > > base in mind and like most IT implementations it got away from us > > quickly and we have over 1000 users now. We are experiencing huge > > slowdowns and lots of complaints of performance. > > > > Let me outline our system: > > > > I have a load balancer that round robins incoming request to 3 zeo servers > > > > I have 1 zope server on the back en that severs the data, this is a > > large single processor machine with 4 gigs of ram > > > > Each zeo server is 2 gigs of ram, single processor 3.2 gighz > > > > The throughput doesn't appear to be the problem as ram/processors are. > > > > Short term I plan to change all zeo's to 4 gigs ram and add 3 additional > > servers. > > > > > > The Technologies we are using are Zope, Zeo, plone and so forth. Why > > are we having these large issues? I see sites with hundreds of > > thousands of users and they don't seem to have these issues? We plan to > > recode our site to get away from the Data.fs as soon as we can but that > > is a ways off yet. What am I doing wrong can anyone help? > > You will need to find out where the slow bits are. There are various > profiling tools for use in Zope. PTProfiler, CallProfiler, etc. Also > examine your ZODB cache graphs and settings. These can tell you where > your bottlenecks are. > > There are lots of ways to improve performance just administratively. You > can improve various zope.conf settings, and set up caching (either RAM > or HTTP Accel.) in Plone. You can also put a caching proxy server (like > Squid) in front of your Zopes. > > Also, Plone 2.1 is significantly faster. > > --jcc > -- > "Building Websites with Plone" > http://plonebook.packtpub.com/ > > Enfold Systems, LLC > http://www.enfoldsystems.com > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope scalabilty and problems
Kennamore, Matthew G [NTK] wrote: Here is our situation, we originally launched zope with a small user base in mind and like most IT implementations it got away from us quickly and we have over 1000 users now. We are experiencing huge slowdowns and lots of complaints of performance. Let me outline our system: I have a load balancer that round robins incoming request to 3 zeo servers I have 1 zope server on the back en that severs the data, this is a large single processor machine with 4 gigs of ram Each zeo server is 2 gigs of ram, single processor 3.2 gighz The throughput doesn’t appear to be the problem as ram/processors are. Short term I plan to change all zeo’s to 4 gigs ram and add 3 additional servers. The Technologies we are using are Zope, Zeo, plone and so forth. Why are we having these large issues? I see sites with hundreds of thousands of users and they don’t seem to have these issues? We plan to recode our site to get away from the Data.fs as soon as we can but that is a ways off yet. What am I doing wrong can anyone help? You will need to find out where the slow bits are. There are various profiling tools for use in Zope. PTProfiler, CallProfiler, etc. Also examine your ZODB cache graphs and settings. These can tell you where your bottlenecks are. There are lots of ways to improve performance just administratively. You can improve various zope.conf settings, and set up caching (either RAM or HTTP Accel.) in Plone. You can also put a caching proxy server (like Squid) in front of your Zopes. Also, Plone 2.1 is significantly faster. --jcc -- "Building Websites with Plone" http://plonebook.packtpub.com/ Enfold Systems, LLC http://www.enfoldsystems.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope scalabilty and problems
On 8/31/05, Kennamore, Matthew G [NTK] <[EMAIL PROTECTED]> wrote: > The Technologies we are using are Zope, Zeo, plone and so forth. Why are we > having these large issues? I see sites with hundreds of thousands of users > and they don't seem to have these issues? We plan to recode our site to get > away from the Data.fs as soon as we can but that is a ways off yet. What am > I doing wrong can anyone help? Always hard to tell, but maybe you do many writes to the ZODB? As long as you only read Zope can be very fast, but writes to the ZODB will take a long time. Also, Plone (and any CMF technology in fact) has a serious performance hit in the skins system. There are some improvements possible there, like giving visitors a special skin that includes fewer skin folders and such. Check on a plone list for more help there. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
