[Zope] Fwd: Re: [Zope] zope2.2.0 and what is high load
conclusion (sort of) > > > I suggest that you: > > > > > > 1) Monitor your Zope clients' thread usages by using their debug > > > facilities. did this and saw that a lot of connections spent on getting images. when the images were moved so that apache serve them, the server is more responsive, now, i seldom encounter more than 100 simultaneous httpd connection. it's as if zope is responding slowly that connections to apache accumulate. all the images are small in size (<20k). i still don't have the answers to why that will affect response time. anybody? and i'm sure i never did any other changes to the setup. only moving images to apache. thanks --- ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] zope2.2.0 and what is high load
Based on your findings that all 7 DB connections are being used, I'd revise the number of threads to 8. will give it a go this monday. Yes, it's the pool_size parameter. You can increment this value *carefully* if you see that you have more threads hanging around than DB connections, like you're seeing now. I have no idea what this will do to your relational stuff. ---that really scares me. sailing the uncharted see, and me a non swimmer! The "right thing to do" is to test this outside of production. I don't want to tell you to go ahead and muck around on a production server. But if it were *me*, I might be tempted to: --like you, i'm tempted too. i'll do backup first before anything else. furthermore, i can't get the amount of traffic to really test. - First, lower the number of threads via -t8 and see what effect that has on speed. - If that doesn't solve anything, bump up both the NUMBER_OF_THREADS and the pool_size up to 10 or 12, then do some checking, and see if you still have threads waiting for a DB. Continue doing this in increments of 2 or 4, making notes as you go. Note that if it breaks your site, I'll deny I ever gave you this advice. :-) Proceed at your own risk. ---thanks for the advice that's not given ;) , i'll report progress monday! Geez, I hope not. This is an eminently solveable problem. --i hope to solve this, and prove the greatness of OSS! ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
in general, when a thread attempts to do anything with zope it needs to get hold of a ZODB connection so it can read or otherwise manipulate objects in the ZODB. this is why the number of application threads should be smaller than the number of ZODB connections. tweaking these numbers is possible, but mileage may vary depending on your specific site and usage. --- as noted by chris M, if i were to bump up the zodn connection pool, it better be more than the app thread, right? so, in my previous case, i increased the app threads to be more than the zodb connection pool. what does this entail? i'm not too sure. i intend to increase the app thread number and the connection pool number, to see whether these increase will have any effeect on my currently loaded site. one tiny question, what will all these do to my rdb connection. my DA is not thread safe, me think. (ZPYgreSql) jens thanks Jens Vagelpohl [EMAIL PROTECTED] Software Engineer www.digicool.com Digital Creations (888) 344-4332 Got Zope? > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On > Behalf Of Dario > Lopez-Kästen > Sent: Friday, August 11, 2000 07:10 > To: [EMAIL PROTECTED] > Subject: [Zope] Defining High Load in Heavily DB Driven site (was: > [Zope] zope2.2.0 and what is high load) > > > Hello! > > I need to find more info on how Zope handles threading with > relation to > database connections. > > I am currently considering and evaluating Zope as one of the > options we have > to build a really large, completely databasedriven "enterprise scale" > web-platform. I am a bit worried about this "maximum of 7 > threads per db > connection" limitation mentioned previously, and I need to > know a) what does > this mean in terms of accesibility, b) how does it affect > performace, and c) > how does Zope work (in detail) with external database connections. > > We are going to be using Oracle as our DB backend, all our > served data will > be database *only* (some of it will be fairly large), and > efficient database > connections is *crucial* to what we intend to do. > > We are going to have between 10-20k users at most and I > expect the maximun > of simultaneous connections to be beween 400 - 900 under > heavy load; "normal > load" will probably oscillate around 25-200 simultaneous > connections (these > numbers are crude guesses, but take into account increased > usage over time > as more services are provided in the future). > > Bandwith problems are not an issue, nor is computing power. > > I would appreciate any pointers to documentation or people to ask. > > Sincerely, > > /dario > ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
Here's my understanding of how it works: - The number of threads is decided by NUMBER_OF_THREADS in z2.py or the command line argument -t[whatever]. - You can give ZServer/Zope a hundred threads if you want to, but if you want more DB connections to service those threads with, you need to change the pool_size in DB.py. Because Dario's site is hypothetical at the moment, it's probably not a good idea to start out by changing these parameters. He should probably just try it out at default, knowing that he can (carefully) bump these values up as necessary if he notices problems. It's sort of pointless to guess about how to increase speed until you know you need it to be increased. Zope can handle this kind of load, it's just a matter of hitting the sweet spot. In order to do that, you need to do some profiling work, unfortunately. > -Original Message- > From: Paul Everitt [mailto:[EMAIL PROTECTED]] > Sent: Friday, August 11, 2000 9:03 AM > To: Dario Lopez-Kästen > Cc: [EMAIL PROTECTED] > Subject: Re: [Zope] Defining High Load in Heavily DB Driven site (was: > [Zope] zope2.2.0 and what is high load) > > > "Dario Lopez-Kästen" wrote: > > > > Hello! > > Hello Dario! > > > I need to find more info on how Zope handles threading with > relation to > > database connections. > > How about: > > http://www.zope.org/Documentation/Developer/Models/ZODB > > The jargon of threads and connections (and then threads again) can be > pretty confusing in Zope. > > First, ZServer is the thing in front listening for socket > requests. It > has a connection pool (select based) which maps into Python threads in > the Zope application. (ZServer is part of the Zope process, I'm > discussing logical partitioning.) > > Each concept of a "thread" in Zope is actually a "database connection" > to the ZODB. Threads get a private copy of the transaction > space, thus > greatly simplifying (to the point of removal) the need for programmers > to think about thread safety. > > If you're talking about the thread concurrency a connection to Oracle, > then that's a bit different. I _think_ that ultimately this > maps to the > number of Zope database connections. > > This conversation is probably better in the zope-dev list. > > > I am currently considering and evaluating Zope as one of > the options we have > > to build a really large, completely databasedriven > "enterprise scale" > > web-platform. I am a bit worried about this "maximum of 7 > threads per db > > This isn't a maximum. It's just the default. It's easy to > increase the > number of connections. > > > connection" limitation mentioned previously, and I need to > know a) what does > > this mean in terms of accesibility, b) how does it affect > performace, and c) > > how does Zope work (in detail) with external database connections. > > > > We are going to be using Oracle as our DB backend, all our > served data will > > be database *only* (some of it will be fairly large), and > efficient database > > connections is *crucial* to what we intend to do. > > Since you're not using any persistent objects in Zope (unless you do > caching of the SQL data, which you probably should), then you > don't have > to worry about process growth from having more database > connections for > Zope. Jack it up to 25. > > Since much of the time in a Zope hit is taken by rendering to HTML, a > number like 25 probably means you can process a hundred simultaneous > Oracle requests. That works out to be an average of around > ten million > hits per day. If you want more, increase the number. > > > We are going to have between 10-20k users at most and I > expect the maximun > > of simultaneous connections to be beween 400 - 900 under > heavy load; "normal > > load" will probably oscillate around 25-200 simultaneous > connections (these > > numbers are crude guesses, but take into account increased > usage over time > > as more services are provided in the future). > > > > Bandwith problems are not an issue, nor is computing power. > > > > I would appreciate any pointers to documentation or people to ask. > > Is this an intranet or a public site? > > --Paul > > ___ > Zope maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) > ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] zope2.2.0 and what is high load
> On Fri, 11 Aug 2000, Chris McDonough wrote: > > > i went to each Zeo client control_panel, and see that there's > > > a mzximum of > > > four connections open at any given time. > > > comments to the above. i saw only four threads because i > started that > particular box with the default. on the box with -t25, i see 7 open > connections at any given time. sorry, my mistake. Oh, ok... that makes more sense. > > You may want to ramp your number of threads down to 5 > (-t5). 25 doesn't > > do much for you because only 7 threads can be using > database connections > > at any given time. The high number of threads you're > servicing might > > actually be slowing things down as they contend for > available database > > connections. > > > will do. and hope to see that it does something. Based on your findings that all 7 DB connections are being used, I'd revise the number of threads to 8. > > It doesn't seem as if bumping your number of database > connections will > > do us any good here, because it doesn't seem you're using > all 7. I'm > > not sure *why* you're not using all 7, because you've got > at least 15 > > processes waiting around just on your top screen for DB connections. > > i did some grepping and saw the magical number 7 in > ZODB/DB.py. is that the > number to turn up if i wanted to try increasing the db pool > connection? i do > have postgresql with ZPygreSQL. will that have anything to > do with what i'm > facing? Yes, it's the pool_size parameter. You can increment this value *carefully* if you see that you have more threads hanging around than DB connections, like you're seeing now. I have no idea what this will do to your relational stuff. The "right thing to do" is to test this outside of production. I don't want to tell you to go ahead and muck around on a production server. But if it were *me*, I might be tempted to: - First, lower the number of threads via -t8 and see what effect that has on speed. - If that doesn't solve anything, bump up both the NUMBER_OF_THREADS and the pool_size up to 10 or 12, then do some checking, and see if you still have threads waiting for a DB. Continue doing this in increments of 2 or 4, making notes as you go. Note that if it breaks your site, I'll deny I ever gave you this advice. :-) Proceed at your own risk. > i love zope, and open source in general, and i can't bear > having to switch to > using something i don't really like. Geez, I hope not. This is an eminently solveable problem. ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
"Dario Lopez-Kästen" wrote: > > Hello! Hello Dario! > I need to find more info on how Zope handles threading with relation to > database connections. How about: http://www.zope.org/Documentation/Developer/Models/ZODB The jargon of threads and connections (and then threads again) can be pretty confusing in Zope. First, ZServer is the thing in front listening for socket requests. It has a connection pool (select based) which maps into Python threads in the Zope application. (ZServer is part of the Zope process, I'm discussing logical partitioning.) Each concept of a "thread" in Zope is actually a "database connection" to the ZODB. Threads get a private copy of the transaction space, thus greatly simplifying (to the point of removal) the need for programmers to think about thread safety. If you're talking about the thread concurrency a connection to Oracle, then that's a bit different. I _think_ that ultimately this maps to the number of Zope database connections. This conversation is probably better in the zope-dev list. > I am currently considering and evaluating Zope as one of the options we have > to build a really large, completely databasedriven "enterprise scale" > web-platform. I am a bit worried about this "maximum of 7 threads per db This isn't a maximum. It's just the default. It's easy to increase the number of connections. > connection" limitation mentioned previously, and I need to know a) what does > this mean in terms of accesibility, b) how does it affect performace, and c) > how does Zope work (in detail) with external database connections. > > We are going to be using Oracle as our DB backend, all our served data will > be database *only* (some of it will be fairly large), and efficient database > connections is *crucial* to what we intend to do. Since you're not using any persistent objects in Zope (unless you do caching of the SQL data, which you probably should), then you don't have to worry about process growth from having more database connections for Zope. Jack it up to 25. Since much of the time in a Zope hit is taken by rendering to HTML, a number like 25 probably means you can process a hundred simultaneous Oracle requests. That works out to be an average of around ten million hits per day. If you want more, increase the number. > We are going to have between 10-20k users at most and I expect the maximun > of simultaneous connections to be beween 400 - 900 under heavy load; "normal > load" will probably oscillate around 25-200 simultaneous connections (these > numbers are crude guesses, but take into account increased usage over time > as more services are provided in the future). > > Bandwith problems are not an issue, nor is computing power. > > I would appreciate any pointers to documentation or people to ask. Is this an intranet or a public site? --Paul ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
hi dario, to clear up some possible misunderstandings and naming issues: we have two different beasts here. one is the number of threads that zope uses while it runs. this can be tweaked by invoking the start script with an argument like "t " or by editing z2.py and changing the default number of 4 threads. again, these are application threads. the database connections that were talked about are a different issue. these refer to the number of connections zope creates to its underlying ZODB object store. you can see them if you go to /Control_Panel/manage_debug in Zope < 2.2.0 or by using the link in the control panel in Zope > 2.2.0. the number of these connections is set in /lib/python/ZODB/DB.py and the default is 7. in general, when a thread attempts to do anything with zope it needs to get hold of a ZODB connection so it can read or otherwise manipulate objects in the ZODB. this is why the number of application threads should be smaller than the number of ZODB connections. tweaking these numbers is possible, but mileage may vary depending on your specific site and usage. jens Jens Vagelpohl [EMAIL PROTECTED] Software Engineer www.digicool.com Digital Creations (888) 344-4332 Got Zope? > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On > Behalf Of Dario > Lopez-Kästen > Sent: Friday, August 11, 2000 07:10 > To: [EMAIL PROTECTED] > Subject: [Zope] Defining High Load in Heavily DB Driven site (was: > [Zope] zope2.2.0 and what is high load) > > > Hello! > > I need to find more info on how Zope handles threading with > relation to > database connections. > > I am currently considering and evaluating Zope as one of the > options we have > to build a really large, completely databasedriven "enterprise scale" > web-platform. I am a bit worried about this "maximum of 7 > threads per db > connection" limitation mentioned previously, and I need to > know a) what does > this mean in terms of accesibility, b) how does it affect > performace, and c) > how does Zope work (in detail) with external database connections. > > We are going to be using Oracle as our DB backend, all our > served data will > be database *only* (some of it will be fairly large), and > efficient database > connections is *crucial* to what we intend to do. > > We are going to have between 10-20k users at most and I > expect the maximun > of simultaneous connections to be beween 400 - 900 under > heavy load; "normal > load" will probably oscillate around 25-200 simultaneous > connections (these > numbers are crude guesses, but take into account increased > usage over time > as more services are provided in the future). > > Bandwith problems are not an issue, nor is computing power. > > I would appreciate any pointers to documentation or people to ask. > > Sincerely, > > /dario > ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
[Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
Hello! I need to find more info on how Zope handles threading with relation to database connections. I am currently considering and evaluating Zope as one of the options we have to build a really large, completely databasedriven "enterprise scale" web-platform. I am a bit worried about this "maximum of 7 threads per db connection" limitation mentioned previously, and I need to know a) what does this mean in terms of accesibility, b) how does it affect performace, and c) how does Zope work (in detail) with external database connections. We are going to be using Oracle as our DB backend, all our served data will be database *only* (some of it will be fairly large), and efficient database connections is *crucial* to what we intend to do. We are going to have between 10-20k users at most and I expect the maximun of simultaneous connections to be beween 400 - 900 under heavy load; "normal load" will probably oscillate around 25-200 simultaneous connections (these numbers are crude guesses, but take into account increased usage over time as more services are provided in the future). Bandwith problems are not an issue, nor is computing power. I would appreciate any pointers to documentation or people to ask. Sincerely, /dario - Dario Lopez-Kästen Systems Developer Chalmers Univ. of Technology [EMAIL PROTECTED] ICQ will yield no hitsIT Systems & Services ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] zope2.2.0 and what is high load
On Fri, 11 Aug 2000, Chris McDonough wrote: > > i went to each Zeo client control_panel, and see that there's > > a mzximum of > > four connections open at any given time. > comments to the above. i saw only four threads because i started that particular box with the default. on the box with -t25, i see 7 open connections at any given time. sorry, my mistake. > > That's about it for your problem. Maybe the profiling stuff can give > more info, I'm not sure. > i'll try and read more about that profiling. but not knowing what it does really scares me :) > It shows that four threads have database connections. I'm curious as to > why you're only seeing four threads being served by database connections > too. as mentioned above, my bad. right after i sent the prev mail, i went to the other box with -t25, and it does have 7 open connections. > > You may want to ramp your number of threads down to 5 (-t5). 25 doesn't > do much for you because only 7 threads can be using database connections > at any given time. The high number of threads you're servicing might > actually be slowing things down as they contend for available database > connections. > will do. and hope to see that it does something. > Why are you frontending this with apache? how are you servicing two ZEO > clients via a single apache? Does it do some sort of round-robining? > Or does each box have its own apache? each box has its own apache. i'm round robining (straight dns round robin). ZEO is also a concern here because i'm using it blindly, unfortunately. but it works great if i got less than 150 simultaneous connections. and this symptom i saw prevuiously, even without ZEO. Is there any reason you're not > just using straight ZServer? I'm wondering if apache processes > contending to talk to available Zope python threads might be slowing > stuff down? we need to serve some othe things from apache, that can't be done thru ZServer alone (a little php stuff) > > It doesn't seem as if bumping your number of database connections will > do us any good here, because it doesn't seem you're using all 7. I'm > not sure *why* you're not using all 7, because you've got at least 15 > processes waiting around just on your top screen for DB connections. i did some grepping and saw the magical number 7 in ZODB/DB.py. is that the number to turn up if i wanted to try increasing the db pool connection? i do have postgresql with ZPygreSQL. will that have anything to do with what i'm facing? > Maybe manage_debug isn't telling you the whole story, or maybe you > caught it at a low-load period. I wonder if the profiling stuff would > help here. > > Maybe someone else can jump in here, this is all I can think of at the > moment. > thanks for sparing some time helping me out. i love zope, and open source in general, and i can't bear having to switch to using something i don't really like. ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] zope2.2.0 and what is high load
> i went to each Zeo client control_panel, and see that there's > a mzximum of > four connections open at any given time. OK, I'm not sure this is real meaningful. A lot can happen in between refreshes. This is only a rough sort of "realtime" guess of what's going on. > why wouldn't increasing the number of threads with -t25 > affect the way zope > behave? do i need to change it in z2.py? No... -t[whatever] does the same thing as NUMBER_OF_THREADS in z2.py. > > There is also a profiling facility inside > Control_Panel/debug that does > > some sort of automated performance compilation for you, though I've > > never had a reason to use it. When it's running, it also > purportedly > > slows Zope down quite a bit. > > > > profiling is not enabled since i dont really know the heck it does. I don't know what it does specifically, either. I've never needed to use it. Maybe somebody else can comment. > > I suggest that you: > > > > 1) Monitor your Zope clients' thread usages by using their debug > > facilities. > done. as mentioned above, the number of connections is four > at most. what > else should i look for in the debugInfo? That's about it for your problem. Maybe the profiling stuff can give more info, I'm not sure. > most of the z2 threads are used, although only 3-4 are used > heavily. what > does this show? It shows that four threads have database connections. I'm curious as to why you're only seeing four threads being served by database connections too. You may want to ramp your number of threads down to 5 (-t5). 25 doesn't do much for you because only 7 threads can be using database connections at any given time. The high number of threads you're servicing might actually be slowing things down as they contend for available database connections. Why are you frontending this with apache? how are you servicing two ZEO clients via a single apache? Does it do some sort of round-robining? Or does each box have its own apache? Is there any reason you're not just using straight ZServer? I'm wondering if apache processes contending to talk to available Zope python threads might be slowing stuff down? It doesn't seem as if bumping your number of database connections will do us any good here, because it doesn't seem you're using all 7. I'm not sure *why* you're not using all 7, because you've got at least 15 processes waiting around just on your top screen for DB connections. Maybe manage_debug isn't telling you the whole story, or maybe you caught it at a low-load period. I wonder if the profiling stuff would help here. Maybe someone else can jump in here, this is all I can think of at the moment. ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] zope2.2.0 and what is high load
On Thu, 10 Aug 2000, Chris McDonough wrote: > It sounds like your load may not be very small. > > Your -t25 setting to threads does not help too much, because the number > of database connections is hard-limited to 7 in the ZODB source. > > It might help to visit the debug section of the control_panel to get an > understanding of how many threads are actually being used at any given > moment. If you find that all 7 threads are being continually used > simultaneously, and your box can handle more threads (e.g. it's not CPU- > or RAM-bound already), we can probably figure out how to bump you up to > more database connection threads per client by changing the source. > This should really be a knob somewhere. > i went to each Zeo client control_panel, and see that there's a mzximum of four connections open at any given time. this is the number of onnections right after update snapshot/refresh /auto refresh statement in control_panel. to confirm, i went to all other zope 2.2.0 instance that i got, - yes, only four spaces available. i remember seing the number 4 is mentioned in z2.py from z2.py # The size of the thread pool, if ZODB3 is used. NUMBER_OF_THREADS=4 ---end CPU and RAM is plenty in one of the boxes (4 G RAM, 2 CPUs) and quite limited in another (700k RAM and 2CPUs). why wouldn't increasing the number of threads with -t25 affect the way zope behave? do i need to change it in z2.py? what are the things that i should know before i do that? will this have effect on connections that access data from rdb? sorry if i sound like a fool here, (maybe i am :)), but i don't get zope that deep yet. > There is also a profiling facility inside Control_Panel/debug that does > some sort of automated performance compilation for you, though I've > never had a reason to use it. When it's running, it also purportedly > slows Zope down quite a bit. > profiling is not enabled since i dont really know the heck it does. > Or, of course, you could always throw in another ZEO client if you're > getting hit too hard to handle the load. > this will be the last resort because it cost us dollars > I suggest that you: > > 1) Monitor your Zope clients' thread usages by using their debug > facilities. done. as mentioned above, the number of connections is four at most. what else should i look for in the debugInfo? > 2) Monitor your Zope clients' general CPU and memory usages via vmstat > or top or what-have you. 9:45am up 21 days, 20:29, 4 users, load average: 1.73, 1.63, 1.59 318 processes: 314 sleeping, 4 running, 0 zombie, 0 stopped CPU states: 109.0% user, 17.0% system, 0.0% nice, 76.9% idle CPU0 states: 51.5% user, 8.2% system, 0.0% nice, 39.2% idle CPU1 states: 57.1% user, 8.3% system, 0.0% nice, 33.5% idle Mem: 776788K av, 770932K used, 5856K free, 580284K shrd, 144312K buff Swap: 134616K av, 1540K used, 133076K free378260K cached PID USER PRI NI SIZE RSS SHARE LC STAT %CPU %MEM TIME COMMAND 8907 nobody18 0 70080 68M 1816 3 R10.9 9.0 50:30 python 8921 nobody 5 0 70080 68M 1816 3 S10.7 9.0 38:24 python 8924 nobody17 0 70080 68M 1816 3 S10.1 9.0 31:33 python 8914 nobody 6 0 70080 68M 1816 2 S 7.6 9.0 45:43 python 8920 nobody11 0 70080 68M 1816 2 S 7.2 9.0 44:23 python 15165 root 17 0 948 948 648 2 R 6.8 0.1 0:03 top 8919 nobody11 0 70080 68M 1816 2 S 5.3 9.0 40:27 python 8930 nobody 4 0 70080 68M 1816 3 S 4.2 9.0 30:31 python 8922 nobody 6 0 70080 68M 1816 2 S 4.0 9.0 36:01 python 8928 nobody12 0 70080 68M 1816 3 S 4.0 9.0 36:49 python 8923 nobody 5 0 70080 68M 1816 3 S 3.8 9.0 36:27 python 8925 nobody10 0 70080 68M 1816 3 S 3.8 9.0 42:51 python 8927 nobody 3 0 70080 68M 1816 3 S 3.8 9.0 35:40 python 8915 nobody 3 0 70080 68M 1816 2 S 3.7 9.0 36:08 python 8917 nobody10 0 70080 68M 1816 2 S 3.7 9.0 39:04 python 8918 nobody 5 0 70080 68M 1816 2 S 3.7 9.0 32:25 python 8931 nobody 5 0 70080 68M 1816 3 S 3.7 9.0 32:34 python 8926 nobody10 0 70080 68M 1816 2 S 3.5 9.0 31:47 python 8932 nobody10 0 70080 68M 1816 3 S 3.5 9.0 34:38 python 8933 nobody10 0 70080 68M 1816 2 S 3.5 9.0 37:53 python 8916 nobody 7 0 70080 68M 1816 2 R 3.1 9.0 41:20 python 13664 postgres 2 0 3696 3696 3204 3 S 1.4 0.4 1:54 postmaster 8561 postgres 2 0 3716 3716 3208 2 S 1.2 0.4 8:06 postmaster 8560 postgres 2 0 3724 3724 3220 2 S 1.1 0.4 8:18 postmaster 8563 postgres 2 0 3740 3740 3212 3 S 0.9 0.4 8:01 postmaster 13825 postgres 2 0 3644 3644 3164 3 S 0.9 0.4 1:00 postmaster ---end top- most of the z2 threads are used, although only 3-4 are used heav
RE: [Zope] zope2.2.0 and what is high load
It sounds like your load may not be very small. Your -t25 setting to threads does not help too much, because the number of database connections is hard-limited to 7 in the ZODB source. It might help to visit the debug section of the control_panel to get an understanding of how many threads are actually being used at any given moment. If you find that all 7 threads are being continually used simultaneously, and your box can handle more threads (e.g. it's not CPU- or RAM-bound already), we can probably figure out how to bump you up to more database connection threads per client by changing the source. This should really be a knob somewhere. There is also a profiling facility inside Control_Panel/debug that does some sort of automated performance compilation for you, though I've never had a reason to use it. When it's running, it also purportedly slows Zope down quite a bit. Or, of course, you could always throw in another ZEO client if you're getting hit too hard to handle the load. I suggest that you: 1) Monitor your Zope clients' thread usages by using their debug facilities. 2) Monitor your Zope clients' general CPU and memory usages via vmstat or top or what-have you. Then report back your findings and we'll see if we can spot a bottleneck. > -Original Message- > From: bak @ kedai [mailto:[EMAIL PROTECTED]] > Sent: Thursday, August 10, 2000 11:02 AM > To: [EMAIL PROTECTED] > Subject: [Zope] zope2.2.0 and what is high load > > > oo zopistas > i have a question. > being a small time site maintainer, i don't quite know how to > classify the > situation i'm having now. > > the site is front ended with apache, with mod_pcgi2. zope > 2.2.0 (running > with -t 25) and rh 6.2 > > how can i know that the load i'm having is a big load? i've > been getting > more than 200 simultaneos connections(ps ax|grep http|wc -l) > and that affect > the response time. > > if the connection is less than 150, then zope's response time > is great. > > my Q: what do you BIG SITE admin have? do you encounter > slow down? i need > some sort of verification that more than 200 simultaneous > connect is really > a big load. i hope not to have to resort to other costly and > proprietery > solutions just because of my ignorance. > > i know table design does affect loading time, and that will > be tackled. > cache is an option that i've looked at but have to disregard > for now since i > can't get ie to behave. i know of the how to and will be > looking at it. > i've scaled the site (was once more than 400 connections at > one time) with > ZEO to two servers. > > i now hope to have insights from yu BIG SITE admins > > looking forward to your input, help, and pointers. i'll try > and document > every thing and put a howto. > > thanks > > > ___ > Zope maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) > ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
[Zope] zope2.2.0 and what is high load
oo zopistas i have a question. being a small time site maintainer, i don't quite know how to classify the situation i'm having now. the site is front ended with apache, with mod_pcgi2. zope 2.2.0 (running with -t 25) and rh 6.2 how can i know that the load i'm having is a big load? i've been getting more than 200 simultaneos connections(ps ax|grep http|wc -l) and that affect the response time. if the connection is less than 150, then zope's response time is great. my Q: what do you BIG SITE admin have? do you encounter slow down? i need some sort of verification that more than 200 simultaneous connect is really a big load. i hope not to have to resort to other costly and proprietery solutions just because of my ignorance. i know table design does affect loading time, and that will be tackled. cache is an option that i've looked at but have to disregard for now since i can't get ie to behave. i know of the how to and will be looking at it. i've scaled the site (was once more than 400 connections at one time) with ZEO to two servers. i now hope to have insights from yu BIG SITE admins looking forward to your input, help, and pointers. i'll try and document every thing and put a howto. thanks ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
