Re: [Zope-dev] How to make Zope fail nicely under high load?
On Thursday 12 February 2004 00:57, you wrote: One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog. I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances. (my apologies if this is obvious) -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn Stabell wrote:
Okay, I'd better explain our situation in more detail, then.
The point is, our template is very dynamic, and our application (a BBS)
is very dynamic as well. We have done one round of optimization
(basically making things as cacheable as possible) and after testing the
scalability now we're going to do another round of optimization,
tweaking the algorithm and removing as many ZCatalog searches as
possible.
If you are optimising for speed, there is nothing wrong with only making
1 catalog query, and then format the page from that. If possible.
That is basically what php base boards do anyway.
Also if the page is that dynamic, and you need that much speed, maybe
you shouldn't use zpt very much. You could generate the most critical
pages from pure Python.
That would allow for a whole different bag tricks, but still be nicer
than php.
Like caching a generated view and then replacing it vith a python
generated board view.
def re_cache(self)
self.layout_cache = self.board_template(self, self.REQUEST)
def index_html(self):
board_view = self.generate_board_view()
return self.layout_cache.replace('%%board_view%%', board_view)
regards Max M
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Toby wrote: One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog. I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances. (my apologies if this is obvious) Okay, for example, since it's a BBS we need to have quick access to: the latest poster the latest posting the latest registered member All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time. Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO. In effect we want to do something like this: def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster def setLatestPoster(self, poster): self._v_latestPoster = poster But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads: http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html Regards, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
using the _v_ variables won't work for this since different threads won't agree on who was latest. Its definitely a bad idea to do queries for these three data points. What about putting them in the temp_folder then you can still use ZEO with a bunch of clients that will all share the same data but you don't need to write the values to disk every time they change. Even putting them in a normal ZODB storage would be a _lot_ faster than doing a query. -EAD On Feb 12, 2004, at 8:19 AM, Bjorn Stabell wrote: Toby wrote: One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog. I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances. (my apologies if this is obvious) Okay, for example, since it's a BBS we need to have quick access to: the latest poster the latest posting the latest registered member All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time. Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO. In effect we want to do something like this: def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster def setLatestPoster(self, poster): self._v_latestPoster = poster But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads: http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html Regards, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Zope should have something like ASP's Response.IsClientConnected method, which would allow app code to manually abort the processing of a request in strategic places if the connection has already been closed. I mentioned this a while back, but never did do anything about it. On Wed, 2004-02-11 at 20:05, Bjorn Stabell wrote: Bjorn wrote: I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them. Casey wrote: Why would they be closed by the client? Two scenarios: 1) The user gets tired of waiting for the page to come up after a few seconds and stopes or reloads it. This is very common if the response time is above a few seconds. 2) After clicking on a link the user discovers another, better link, and quickly clicks on that one as well. This is quite common for me, at least :) When Zope gets very loaded, response times suffer and these things happen more often; in our case with 300 requests in the pending queue, nobody would bother waiting 10 minutes for their page to come up (they'd stop the page loading and go somewhere else), so all the requests (most likely) would be needless; Zope is in fact just processing dead requests. Bye, ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Erik: using the _v_ variables won't work for this since different threads won't agree on who was latest. Its definitely a bad idea to do queries for these three data points. What about putting them in the temp_folder then you can still use ZEO with a bunch of clients that will all share the same data but you don't need to write the values to disk every time they change. Even putting them in a normal ZODB storage would be a _lot_ faster than doing a query. You're right, _v_ wouldn't work. So with temp_folder (TemporaryStorage) there is basically no need for the SharedResource product anymore? For ZEO, my understanding is that we would still have to store the values in the ZODB, though. Regards, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn Stabell wrote: (..)For ZEO, my understanding is that we would still have to store the values in the ZODB, though. You are able to store *and* share data over ZEO Clients in a TemporaryStorage by making the ZSS serve this TemporaryStorage to its ZCs. You could mount this storage as temp_folder I guess, but any other mount point would work too. Not sure, but I'd expect so, whether this has significant perfomance advantages over a 'normal' ZODB served by the ZSS. regards, jw -- Jan-Wijbrand Kolman [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Jan-Wijbrand wrote: [...] You are able to store *and* share data over ZEO Clients in a TemporaryStorage by making the ZSS serve this TemporaryStorage to its ZCs. You could mount this storage as temp_folder I guess, but any other mount point would work too. Col :) Not sure, but I'd expect so, whether this has significant perfomance advantages over a 'normal' ZODB served by the ZSS. Probably, and it's no-packing as well. Very nice. We'll definitely use this then. Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
On Thu, 2004-02-12 at 07:19, Bjorn Stabell wrote: Toby wrote: One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog. I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances. (my apologies if this is obvious) Okay, for example, since it's a BBS we need to have quick access to: the latest poster the latest posting the latest registered member Create a zope product that doesn't keep a history (otherwise the ZODB will grow like nuts) and store these values in it whenever there is a new port/registration. Then just read them when you need to. Anyone see anything wrong with that approach? All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time. Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO. In effect we want to do something like this: def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster def setLatestPoster(self, poster): self._v_latestPoster = poster But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads: http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html Regards, -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Thu, 2004-02-12 at 10:06, Jan-Wijbrand Kolman wrote: Bjorn Stabell wrote: (..)For ZEO, my understanding is that we would still have to store the values in the ZODB, though. You are able to store *and* share data over ZEO Clients in a TemporaryStorage by making the ZSS serve this TemporaryStorage to its ZCs. You could mount this storage as temp_folder I guess, but any other mount point would work too. Cool, didn't know that. Not sure, but I'd expect so, whether this has significant perfomance advantages over a 'normal' ZODB served by the ZSS. regards, jw -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] How to make Zope fail nicely under high load?
Hi, We've run into an interesting problem when load-testing a Zope site behind Apache, a problem that we've also encountered in real life. Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? The ideal would be for requests to fail immediately if the backlog of work is more than a certain number of requests (or even better, estimated time to process). Here's what we've tried: Naively, we thought we could just set the socket.listen() backlog in Apache and Zope to a lower value, but in TCP connect()'s apparently don't fail if the server's listen backlog is full; instead requests are retried, resulting in a client side managed listen backlog, also giving the same long latency. (If someone knows this stuff, please confirm/deny these allegations against TCP :) It appears the way to control it would for Apache or Zope to return 503 Service Unavailable when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers would, but that's a bit overkill since we run the server on one machine. Regards, -- Bjorn Stabell [EMAIL PROTECTED] Tel +86 (10) 65918490 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn Stabell wrote: It appears the way to control it would for Apache or Zope to return 503 Service Unavailable when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers would, but that's a bit overkill since we run the server on one machine. Hi, I'm very interested in your resutsl. We have exaclty the same situation, though we have not analysed it further - the only thing that saves us is to restart zope. re: load balancing: there is Pound - http://www.apsis.ch/pound/ - that might serve as a loadbalancer between processes in a single machine. Maybe that will help a bit, though it doesn't really resolve the issue with zope choking; if it works, it will merely circumvent it. hth, /dario -- -- --- Dario Lopez-Kästen, IT Systems Services Chalmers University of Tech. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn Stabell wrote: Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? Make the app faster so that doesn't happen. The ideal would be for requests to fail immediately if the backlog of work is more than a certain number of requests (or even better, estimated time to process). Kinda... It appears the way to control it would for Apache or Zope to return 503 Service Unavailable when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers would, but that's a bit overkill since we run the server on one machine. ...return 503 to every new request probably isn't a good idea, 503 to every new session is probably OK. Maybe. It depends on the workload. If you have a user who sends 10 requests and 5 of them fail, they're going to keep hitting reload to try to get them all to work, which just compounds your problems. The idea is that you want tweak it so some of your users get everything, and some get nothing, instead of everybody getting partials. This means you have to track sessions though... thats probably easier to do in ZServer than it is in Apache 1.3 just because of the process model, but that doesn't mean its easy. By default, Apache doesn't track sessions and do that kind of planned failure. There are probably modules that can enable that behavior. ZServer's multiplexed IO model just hides the accepting socket from the poller if its concurrency level is reached, and yeah, then the incoming connections end up in the backlog. You'd have to change ZServer so it handled those new requests instead, identified if they were part of a session, queued them up if they were, or spat out a 503 if they weren't. Frankly, I bet you'd have more fun just making your app faster. -- Jamie Heilman http://audible.transient.net/~jamie/ Paranoia is a disease unto itself, and may I add, the person standing next to you may not be who they appear to be, so take precaution. -Sathington Willoughby ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn Stabell wrote: Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? Make the app faster so that doesn't happen. You'll always reach the bottleneck sooner or later, and in 99% of the cases it'll be Zope. It's not exactly a racing horse. [...] ...return 503 to every new request probably isn't a good idea, 503 to every new session is probably OK. Maybe. It depends on the workload. If you have a user who sends 10 requests and 5 of them fail, they're going to keep hitting reload to try to get them all to work, which just compounds your problems. The idea is that you want tweak it so some of your users get everything, and some get nothing, instead of everybody getting partials. This means you have to track sessions though... thats probably easier to do in ZServer than it is in Apache 1.3 just because of the process model, but that doesn't mean its easy. By default, Apache doesn't track sessions and do that kind of planned failure. There are probably modules that can enable that behavior. ZServer's multiplexed IO model just hides the accepting socket from the poller if its concurrency level is reached, and yeah, then the incoming connections end up in the backlog. You'd have to change ZServer so it handled those new requests instead, identified if they were part of a session, queued them up if they were, or spat out a 503 if they weren't. With KeepAlive on, wouldn't everything in one session sent over one TCP connection? If that is the case, each accept() will basically service a whole session, and so you should be able to do the easy thing of refusing a connection just after an accept(). Frankly, I bet you'd have more fun just making your app faster. We'll do that as well, but this problem is pretty serious; we can't overload our server for even just a little while before latencies start building up to ridiculous levels and users start hitting reload, further compounding the problem. In the end we have to restart the server, same as Dario reported. Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Wednesday 11 February 2004 09:16, Jamie Heilman wrote: Bjorn Stabell wrote: Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog. You might get some benefit from capping the length of that queue. The idea is that you want tweak it so some of your users get everything, and some get nothing, instead of everybody getting partials. This means you have to track sessions though... Before creating a session, check the size of the ZRendezvous backlog. might work. -- Toby Dickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn Stabell wrote: You'll always reach the bottleneck sooner or later, and in 99% of the cases it'll be Zope. It's not exactly a racing horse. Tell me about it, in a former life I was the guy with the pager. With KeepAlive on, wouldn't everything in one session sent over one TCP connection? If that is the case, each accept() will basically service a whole session, and so you should be able to do the easy thing of refusing a connection just after an accept(). No, a session isn't that nice and tidy, clients can and do open multiple simultaneous connections, connection re-use and pipelining aren't all that define a session. If it were that easy, ZServer would do this already, so would, I imagine, Apache. Its not a pretty problem, which is why I suggest you spend your time looking at what you can do to scale better, or look for an Apache module that implements some sane-sounding session heuristic and give it a whirl, it will increase the resources needed by Apache, but maybe the costs will balance out. We'll do that as well, but this problem is pretty serious; we can't overload our server for even just a little while before latencies start building up to ridiculous levels and users start hitting reload, further compounding the problem. In the end we have to restart the server, same as Dario reported. You'll have to move to load spreading, zeo, etc if you can't tweak your app any further. I personally feel that the ZEO seperation should be compulsary anyway, it just makes more sense, but I digress. Find yourself an Apache module that can spit out 503s, then work on load balancing infrastructure, which is probably the most viable longterm solution (next to simply not using Zope for something it evidently isn't very good at). -- Jamie Heilman http://audible.transient.net/~jamie/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Toby Dickenson wrote: Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog. You might get some benefit from capping the length of that queue. Thats right! Yeah, true, it probably is holding the backlog as the default concurrency (on unix) is pretty high, but capping that queue ... I dunno, what do you with the new requests once its full? You're back to the problem of identifying sessions, and thats a potential mess. Before creating a session, check the size of the ZRendezvous backlog. might work. Yeah, thats a good plan in terms of where to instrument it, if you had to. If it were me, I'd sooner throw more hardware at it than use session identifiers though. -- Jamie Heilman http://audible.transient.net/~jamie/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
*Scratches head*. Isn't it possible to just timestamp every request in a list of running requests, and in the beginning of each request check how long the currently processing requests have been running, and if that sum is above a specified time, fail? OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? Hysteresis could also be used, so that all requests fail after the time limit have been reached once, until the total time fall under another smaller time limit. http://www.homestarrunner.com/systemisdown.html ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Toby wrote: [...] Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog. You might get some benefit from capping the length of that queue. Thanks Toby. It doesn't seem like that queue (the requests list) has a cap at all. Have been stresstesting Zope for a few minutes and the queue is up to 300+ requests in the backlog queue. Not sure how to force a 503 return from within ZRendezvous.handle(), though; that's ZServer medusa magic. 300 requests at 1 request per 2 seconds yields a backlog of work of 10 minutes. This is before we've started optimizing. I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them. Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Toby wrote: Hmm, yes. ZRendezvous doesnt have any choice but to queue up the request. Handling the request at that point could cause problems because the support for pipelined http requests might cause another recursive call to ZRendezvous.handle, and deadlock. You need to handle it earlier. Maybe tweak the readable() method of zhttp_server and zhttp_channel in HTTPServer.py, to stop bytes coming in over the socket if the queue is too large. ? Hm. And making this limit (and the listen backlog parameter) available as configuration options. :) Not letting bytes through is not enough, though. I think a 503 error should be returned as quickly as possible back to the client. I wonder if Zope also processes requests that have been closed from the client end? It will I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them. That doesnt need fixing. It wouldnt be a problem if the queue was kept small. Well, I think it would help in many cases. Many people click on links before the server has finished (or before it has even visibly started) serving up pages. Searching the web today I saw a commercial product (I think it was a load balancer) promoting this as an important feature. Any page for which the client has closed the connection before the server has started serving up the page could be skipped. If the server has already started serving up the page, it could complete the transaction. Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Wed, 11 Feb 2004 20:25:14 +0800 Bjorn Stabell [EMAIL PROTECTED] wrote: Toby wrote: [...] Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog. You might get some benefit from capping the length of that queue. Thanks Toby. It doesn't seem like that queue (the requests list) has a cap at all. Have been stresstesting Zope for a few minutes and the queue is up to 300+ requests in the backlog queue. Not sure how to force a 503 return from within ZRendezvous.handle(), though; that's ZServer medusa magic. 300 requests at 1 request per 2 seconds yields a backlog of work of 10 minutes. This is before we've started optimizing. What kinds of requests are these? Do they all require a dynamic output? If not, then you should put better caching in front of Zope, or at a minimum tweak you caching headers so that some could be served as 304s for instance. If they are all dynamic, how dynamic? How different is the page for one session than another? If much of the page is the same then you might benefit from an ESI approach where you cache parts of pages and assemble them in the cache, serving only small pieces from Zope. I'm a bit surprised that you have not (from what I can see), used the most common Zope speedup (and often the cheapest overall): Buy more hardware and use ZEO. I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them. Why would they be closed by the client? -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
This very much depends on your application and requirements (and your definition of nicely :-)), but I'd argue that it rarely make sense to handle this at the TCP Connection level (just to think about browsers opening multiple connections, HTTP/1.0 or /1.1 compliant browsers, proxies etc.) As an example, one client of ours had the following requirements wrt. to this problem which I think should be fairly common: * Allow X logged-in users till a certain responsiveness threshold is reached * If said threshold is reached: - inform all users trying to log that the site is too loaded - while allowing already-logged-in users to still use the site with acceptable speed This means we had to a) measure responsiveness (which we did with some cobbled together heuristic involving the rate of user logins (something like 5 users per minute) and an external script effectively HTTP Pinging the site every X minutes) and b) redirecting users to a statically hosted page (which can be served cheaply) if said conditions were met. This kind of thing clearly cannot be done at the TCP level because TCP Connection != User session cheers, peter. Bjorn Stabell wrote: Hi, We've run into an interesting problem when load-testing a Zope site behind Apache, a problem that we've also encountered in real life. Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? The ideal would be for requests to fail immediately if the backlog of work is more than a certain number of requests (or even better, estimated time to process). Here's what we've tried: Naively, we thought we could just set the socket.listen() backlog in Apache and Zope to a lower value, but in TCP connect()'s apparently don't fail if the server's listen backlog is full; instead requests are retried, resulting in a client side managed listen backlog, also giving the same long latency. (If someone knows this stuff, please confirm/deny these allegations against TCP :) It appears the way to control it would for Apache or Zope to return 503 Service Unavailable when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers would, but that's a bit overkill since we run the server on one machine. Regards, smime.p7s Description: S/MIME Cryptographic Signature ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn Stabell wrote: Hi, We've run into an interesting problem when load-testing a Zope site behind Apache, a problem that we've also encountered in real life. Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? The ideal would be for requests to fail immediately if the backlog of work is more than a certain number of requests (or even better, estimated time to process). Here's what we've tried: Naively, we thought we could just set the socket.listen() backlog in Apache and Zope to a lower value, but in TCP connect()'s apparently don't fail if the server's listen backlog is full; instead requests are retried, resulting in a client side managed listen backlog, also giving the same long latency. (If someone knows this stuff, please confirm/deny these allegations against TCP :) It appears the way to control it would for Apache or Zope to return 503 Service Unavailable when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers would, but that's a bit overkill since we run the server on one machine. Regards, Naturally, It would be really nice to have a solution inside zope where you can configure stuff like the maxinum number of concurrent sessions, and a hook for handling the refusal of a new session. This so called graceful degradation functionality could also be used for -) graceful shutdown: now, a shutdown of a zope server is brutal. For some applications you want people to be able to finish their business, while at the same time you want that no new users can get into the system. -) licensing policies: having web services that are used by at most X people at the same time. These things have to be handled at the Zope level, since neither the apache, nor the networking layer have any clue about things like user session, aso. Ok, This said. who writes the proposal? ;) Romain Slootmaekers. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly making the problem worse. If the images are fluf, and the user knows they are fluf, then *maybe* they won't reload, and your pages will simply appear ugly. But then you have to ask yourself, why am I sending fluf images that degrades the overall user experience of my application? Clearly there's more optimization that could be done to your application. -- Jamie Heilman http://audible.transient.net/~jamie/ You came all this way, without saying squat, and now you're trying to tell me a '56 Chevy can beat a '47 Buick in a dead quarter mile? I liked you better when you weren't saying squat kid. -Buddy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Wed, 2004-02-11 at 13:09, Jamie Heilman wrote: Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly making the problem worse. If the images are fluf, and the user knows they are fluf, then *maybe* they won't reload, and your pages will simply appear ugly. But then you have to ask yourself, why am I sending fluf images that degrades the overall user experience of my application? Clearly there's more optimization that could be done to your application. Without knowing the app I'd probably say there are many ways that it can be made more efficient. For instance are your image urls being generated using Zope's acquisition? If so, don't do that. Another alternative is to move the static/fluf images out of zope and just let apache serve them. That's just small thing that you can do to reduce the load on zope. Things like squid, zeo, better hardware, re-analyze long running routines, etc are all possible without having to do the other gymnastics that have been discussed. -- Edward Muller - http://www.interlix.com - Open Source Specialists Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network PC Service Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: [EMAIL PROTECTED] - AIM: edwardam453 - ICQ: 287033 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] How to make Zope fail nicely under high load?
On Thursday 12 February 2004 01:23, Casey Duncan wrote: What kinds of requests are these? Do they all require a dynamic output? If not, then you should put better caching in front of Zope, or at a minimum tweak you caching headers so that some could be served as 304s for instance. If they are all dynamic, how dynamic? How different is the page for one session than another? If much of the page is the same then you might benefit from an ESI approach where you cache parts of pages and assemble them in the cache, serving only small pieces from Zope. I'm a bit surprised that you have not (from what I can see), used the most common Zope speedup (and often the cheapest overall): Buy more hardware and use ZEO. I was going to suggest these things too. And a bunch of other stuff. A while ago, I wrote a page about making Zope go faster - it might help. http://zope.org/Members/richard/docs/zope_optimisation.html Richard pgp0.pgp Description: signature ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Richard Jones wrote: On Thursday 12 February 2004 01:23, Casey Duncan wrote: What kinds of requests are these? Do they all require a dynamic output? If not, then you should put better caching in front of Zope, or at a minimum tweak you caching headers so that some could be served as 304s for instance. If they are all dynamic, how dynamic? How different is the page for one session than another? If much of the page is the same then you might benefit from an ESI approach where you cache parts of pages and assemble them in the cache, serving only small pieces from Zope. I'm a bit surprised that you have not (from what I can see), used the most common Zope speedup (and often the cheapest overall): Buy more hardware and use ZEO. I was going to suggest these things too. And a bunch of other stuff. A while ago, I wrote a page about making Zope go faster - it might help. http://zope.org/Members/richard/docs/zope_optimisation.html Okay, I'd better explain our situation in more detail, then. Richard, Casey, thanks. Except for using ZEO, we're already doing everything Richard and you mentions. Zope only serves pages (no images etc) that are dynamic, the object cache is set at 20,000 objects with only 2 threads (otherwise we got too much deactivation of objects), and we also use the RAM Cache Manager for static bits of dynamic pages. We could probably have used it more, but since it can't cache persistent objects, it's a bit of a pain sometimes. The point is, our template is very dynamic, and our application (a BBS) is very dynamic as well. We have done one round of optimization (basically making things as cacheable as possible) and after testing the scalability now we're going to do another round of optimization, tweaking the algorithm and removing as many ZCatalog searches as possible. One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. I'm sure we can make the application run faster, but we'd like to support 500-1000 simultaneous users whereas now we can only support 20-25! I don't know if we can scale it enough just by tweaking the application. Another option is to scrap Zope for the BBS part and use one of the fully featured, and optimized, BBS applications for PHP. That would definitely be a safer choice, but would require extra application integration time (user database, searching). Regards, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Bjorn wrote: I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them. Casey wrote: Why would they be closed by the client? Two scenarios: 1) The user gets tired of waiting for the page to come up after a few seconds and stopes or reloads it. This is very common if the response time is above a few seconds. 2) After clicking on a link the user discovers another, better link, and quickly clicks on that one as well. This is quite common for me, at least :) When Zope gets very loaded, response times suffer and these things happen more often; in our case with 300 requests in the pending queue, nobody would bother waiting 10 minutes for their page to come up (they'd stop the page loading and go somewhere else), so all the requests (most likely) would be needless; Zope is in fact just processing dead requests. Bye, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] How to make Zope fail nicely under high load?
Jamie writes: Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly making the problem worse. If the images are fluf, and the user knows they are fluf, then *maybe* they won't reload, and your pages will simply appear ugly. But then you have to ask yourself, why am I sending fluf images that degrades the overall user experience of my application? Clearly there's more optimization that could be done to your application. If you run Apache as a caching reverse proxy (caching surrogate server) then images will be served from Apache. Only the dynamic HTML pages are served from Zope, so having them fail without regards to sessions is relatively okay. Images and other static content are almost guaranteed to load correctly, and if they didn't, a straight reload from the client will not reload cached images. Regards, -- Bjorn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
