RE: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
hi dario, to clear up some possible misunderstandings and naming issues: we have two different beasts here. one is the number of threads that zope uses while it runs. this can be tweaked by invoking the start script with an argument like "t number_of_threads" or by editing z2.py and changing the default number of 4 threads. again, these are application threads. the database connections that were talked about are a different issue. these refer to the number of connections zope creates to its underlying ZODB object store. you can see them if you go to myzope/Control_Panel/manage_debug in Zope 2.2.0 or by using the link in the control panel in Zope 2.2.0. the number of these connections is set in zope/lib/python/ZODB/DB.py and the default is 7. in general, when a thread attempts to do anything with zope it needs to get hold of a ZODB connection so it can read or otherwise manipulate objects in the ZODB. this is why the number of application threads should be smaller than the number of ZODB connections. tweaking these numbers is possible, but mileage may vary depending on your specific site and usage. jens Jens Vagelpohl [EMAIL PROTECTED] Software Engineer www.digicool.com Digital Creations (888) 344-4332 Got Zope? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Dario Lopez-Kästen Sent: Friday, August 11, 2000 07:10 To: [EMAIL PROTECTED] Subject: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load) Hello! I need to find more info on how Zope handles threading with relation to database connections. I am currently considering and evaluating Zope as one of the options we have to build a really large, completely databasedriven "enterprise scale" web-platform. I am a bit worried about this "maximum of 7 threads per db connection" limitation mentioned previously, and I need to know a) what does this mean in terms of accesibility, b) how does it affect performace, and c) how does Zope work (in detail) with external database connections. We are going to be using Oracle as our DB backend, all our served data will be database *only* (some of it will be fairly large), and efficient database connections is *crucial* to what we intend to do. We are going to have between 10-20k users at most and I expect the maximun of simultaneous connections to be beween 400 - 900 under heavy load; "normal load" will probably oscillate around 25-200 simultaneous connections (these numbers are crude guesses, but take into account increased usage over time as more services are provided in the future). Bandwith problems are not an issue, nor is computing power. I would appreciate any pointers to documentation or people to ask. Sincerely, /dario ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
"Dario Lopez-Kästen" wrote: Hello! Hello Dario! I need to find more info on how Zope handles threading with relation to database connections. How about: http://www.zope.org/Documentation/Developer/Models/ZODB The jargon of threads and connections (and then threads again) can be pretty confusing in Zope. First, ZServer is the thing in front listening for socket requests. It has a connection pool (select based) which maps into Python threads in the Zope application. (ZServer is part of the Zope process, I'm discussing logical partitioning.) Each concept of a "thread" in Zope is actually a "database connection" to the ZODB. Threads get a private copy of the transaction space, thus greatly simplifying (to the point of removal) the need for programmers to think about thread safety. If you're talking about the thread concurrency a connection to Oracle, then that's a bit different. I _think_ that ultimately this maps to the number of Zope database connections. This conversation is probably better in the zope-dev list. I am currently considering and evaluating Zope as one of the options we have to build a really large, completely databasedriven "enterprise scale" web-platform. I am a bit worried about this "maximum of 7 threads per db This isn't a maximum. It's just the default. It's easy to increase the number of connections. connection" limitation mentioned previously, and I need to know a) what does this mean in terms of accesibility, b) how does it affect performace, and c) how does Zope work (in detail) with external database connections. We are going to be using Oracle as our DB backend, all our served data will be database *only* (some of it will be fairly large), and efficient database connections is *crucial* to what we intend to do. Since you're not using any persistent objects in Zope (unless you do caching of the SQL data, which you probably should), then you don't have to worry about process growth from having more database connections for Zope. Jack it up to 25. Since much of the time in a Zope hit is taken by rendering to HTML, a number like 25 probably means you can process a hundred simultaneous Oracle requests. That works out to be an average of around ten million hits per day. If you want more, increase the number. We are going to have between 10-20k users at most and I expect the maximun of simultaneous connections to be beween 400 - 900 under heavy load; "normal load" will probably oscillate around 25-200 simultaneous connections (these numbers are crude guesses, but take into account increased usage over time as more services are provided in the future). Bandwith problems are not an issue, nor is computing power. I would appreciate any pointers to documentation or people to ask. Is this an intranet or a public site? --Paul ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
Here's my understanding of how it works: - The number of threads is decided by NUMBER_OF_THREADS in z2.py or the command line argument -t[whatever]. - You can give ZServer/Zope a hundred threads if you want to, but if you want more DB connections to service those threads with, you need to change the pool_size in DB.py. Because Dario's site is hypothetical at the moment, it's probably not a good idea to start out by changing these parameters. He should probably just try it out at default, knowing that he can (carefully) bump these values up as necessary if he notices problems. It's sort of pointless to guess about how to increase speed until you know you need it to be increased. Zope can handle this kind of load, it's just a matter of hitting the sweet spot. In order to do that, you need to do some profiling work, unfortunately. -Original Message- From: Paul Everitt [mailto:[EMAIL PROTECTED]] Sent: Friday, August 11, 2000 9:03 AM To: Dario Lopez-Kästen Cc: [EMAIL PROTECTED] Subject: Re: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load) "Dario Lopez-Kästen" wrote: Hello! Hello Dario! I need to find more info on how Zope handles threading with relation to database connections. How about: http://www.zope.org/Documentation/Developer/Models/ZODB The jargon of threads and connections (and then threads again) can be pretty confusing in Zope. First, ZServer is the thing in front listening for socket requests. It has a connection pool (select based) which maps into Python threads in the Zope application. (ZServer is part of the Zope process, I'm discussing logical partitioning.) Each concept of a "thread" in Zope is actually a "database connection" to the ZODB. Threads get a private copy of the transaction space, thus greatly simplifying (to the point of removal) the need for programmers to think about thread safety. If you're talking about the thread concurrency a connection to Oracle, then that's a bit different. I _think_ that ultimately this maps to the number of Zope database connections. This conversation is probably better in the zope-dev list. I am currently considering and evaluating Zope as one of the options we have to build a really large, completely databasedriven "enterprise scale" web-platform. I am a bit worried about this "maximum of 7 threads per db This isn't a maximum. It's just the default. It's easy to increase the number of connections. connection" limitation mentioned previously, and I need to know a) what does this mean in terms of accesibility, b) how does it affect performace, and c) how does Zope work (in detail) with external database connections. We are going to be using Oracle as our DB backend, all our served data will be database *only* (some of it will be fairly large), and efficient database connections is *crucial* to what we intend to do. Since you're not using any persistent objects in Zope (unless you do caching of the SQL data, which you probably should), then you don't have to worry about process growth from having more database connections for Zope. Jack it up to 25. Since much of the time in a Zope hit is taken by rendering to HTML, a number like 25 probably means you can process a hundred simultaneous Oracle requests. That works out to be an average of around ten million hits per day. If you want more, increase the number. We are going to have between 10-20k users at most and I expect the maximun of simultaneous connections to be beween 400 - 900 under heavy load; "normal load" will probably oscillate around 25-200 simultaneous connections (these numbers are crude guesses, but take into account increased usage over time as more services are provided in the future). Bandwith problems are not an issue, nor is computing power. I would appreciate any pointers to documentation or people to ask. Is this an intranet or a public site? --Paul ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load)
in general, when a thread attempts to do anything with zope it needs to get hold of a ZODB connection so it can read or otherwise manipulate objects in the ZODB. this is why the number of application threads should be smaller than the number of ZODB connections. tweaking these numbers is possible, but mileage may vary depending on your specific site and usage. --- as noted by chris M, if i were to bump up the zodn connection pool, it better be more than the app thread, right? so, in my previous case, i increased the app threads to be more than the zodb connection pool. what does this entail? i'm not too sure. i intend to increase the app thread number and the connection pool number, to see whether these increase will have any effeect on my currently loaded site. one tiny question, what will all these do to my rdb connection. my DA is not thread safe, me think. (ZPYgreSql) jens thanks Jens Vagelpohl [EMAIL PROTECTED] Software Engineer www.digicool.com Digital Creations (888) 344-4332 Got Zope? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Dario Lopez-Kästen Sent: Friday, August 11, 2000 07:10 To: [EMAIL PROTECTED] Subject: [Zope] Defining High Load in Heavily DB Driven site (was: [Zope] zope2.2.0 and what is high load) Hello! I need to find more info on how Zope handles threading with relation to database connections. I am currently considering and evaluating Zope as one of the options we have to build a really large, completely databasedriven "enterprise scale" web-platform. I am a bit worried about this "maximum of 7 threads per db connection" limitation mentioned previously, and I need to know a) what does this mean in terms of accesibility, b) how does it affect performace, and c) how does Zope work (in detail) with external database connections. We are going to be using Oracle as our DB backend, all our served data will be database *only* (some of it will be fairly large), and efficient database connections is *crucial* to what we intend to do. We are going to have between 10-20k users at most and I expect the maximun of simultaneous connections to be beween 400 - 900 under heavy load; "normal load" will probably oscillate around 25-200 simultaneous connections (these numbers are crude guesses, but take into account increased usage over time as more services are provided in the future). Bandwith problems are not an issue, nor is computing power. I would appreciate any pointers to documentation or people to ask. Sincerely, /dario ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
