[Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Bjorn Stabell
Hi, We've run into an interesting problem when load-testing a Zope site behind Apache, a problem that we've also encountered in real life. Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes.

[Zope-dev] Zope, forking and cpu-affinity

2004-02-11 Thread Dario Lopez-Ksten
Hello, I hope this is not the wrong list... I have some trouble doing cpu-afiinity wiht zope. I can affinitise the parent process just fine, however, zope forks and the children get spread out all over the CPUs. I do not yet wish to resort to shell-hacking the ouput of ps -ef, so I was

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Dario Lopez-Kästen
Bjorn Stabell wrote: It appears the way to control it would for Apache or Zope to return 503 Service Unavailable when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Jamie Heilman
Bjorn Stabell wrote: Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big?

RE: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Bjorn Stabell
Bjorn Stabell wrote: Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Toby Dickenson
On Wednesday 11 February 2004 09:16, Jamie Heilman wrote: Bjorn Stabell wrote: Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Jamie Heilman
Bjorn Stabell wrote: You'll always reach the bottleneck sooner or later, and in 99% of the cases it'll be Zope. It's not exactly a racing horse. Tell me about it, in a former life I was the guy with the pager. With KeepAlive on, wouldn't everything in one session sent over one TCP

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Jamie Heilman
Toby Dickenson wrote: Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog. You might get some benefit from capping the length of

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Lennart Regebro
*Scratches head*. Isn't it possible to just timestamp every request in a list of running requests, and in the beginning of each request check how long the currently processing requests have been running, and if that sum is above a specified time, fail? OK, you get the problem that images may not

RE: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Bjorn Stabell
Toby wrote: [...] Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog. You might get some benefit from capping the length of that

RE: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Bjorn Stabell
Toby wrote: Hmm, yes. ZRendezvous doesnt have any choice but to queue up the request. Handling the request at that point could cause problems because the support for pipelined http requests might cause another recursive call to ZRendezvous.handle, and deadlock. You need to handle it

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Casey Duncan
On Wed, 11 Feb 2004 20:25:14 +0800 Bjorn Stabell [EMAIL PROTECTED] wrote: Toby wrote: [...] Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Peter Sabaini
This very much depends on your application and requirements (and your definition of nicely :-)), but I'd argue that it rarely make sense to handle this at the TCP Connection level (just to think about browsers opening multiple connections, HTTP/1.0 or /1.1 compliant browsers, proxies etc.) As

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Romain Slootmaekers
Bjorn Stabell wrote: Hi, We've run into an interesting problem when load-testing a Zope site behind Apache, a problem that we've also encountered in real life. Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Jamie Heilman
Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly making the problem worse. If the images are fluf, and the

[Zope-dev] Persistence Module Confustion (fwd)

2004-02-11 Thread D. Bickle
I am trying to import the MailHost module that has dependencies with the Persistence package. The interpreter complains that it cannot load the Persistence modules and when I examine the package there is no content or modules. How can I fix my code to resolve these dependencies? Have the

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Edward Muller
On Wed, 2004-02-11 at 13:09, Jamie Heilman wrote: Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly

Re: [Zope-dev] Zope, forking and cpu-affinity

2004-02-11 Thread Dieter Maurer
Dario Lopez-Kästen wrote at 2004-2-11 09:39 +0100: I have some trouble doing cpu-afiinity wiht zope. I can affinitise the parent process just fine, however, zope forks and the children get spread out all over the CPUs. Zope itself does not fork (unless you use system, popen or things like that;

Re: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Richard Jones
On Thursday 12 February 2004 01:23, Casey Duncan wrote: What kinds of requests are these? Do they all require a dynamic output? If not, then you should put better caching in front of Zope, or at a minimum tweak you caching headers so that some could be served as 304s for instance. If they

[Zope-dev] Put an adapted object in context

2004-02-11 Thread Santi Camps
Hi all, Following your instructions and seeing Skin Product code, I've been able to write a simple way to register Apdaters in my applications that are applied to specific meta_type objects during publishing. This is now working fine. I can write an Adapter in my application and say use this

RE: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Bjorn Stabell
Richard Jones wrote: On Thursday 12 February 2004 01:23, Casey Duncan wrote: What kinds of requests are these? Do they all require a dynamic output? If not, then you should put better caching in front of Zope, or at a minimum tweak you caching headers so that some could be served as

RE: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Bjorn Stabell
Bjorn wrote: I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would

RE: [Zope-dev] How to make Zope fail nicely under high load?

2004-02-11 Thread Bjorn Stabell
Jamie writes: Lennart Regebro wrote: OK, you get the problem that images may not load even if the main page does, but is that really worse for the end user than not getting anything? As I've been saying, if you do that, they will reload repeatedly making the problem worse. If the