mod_proxy and balancer problems
Hi,
I am in the process of evaluating Apache 2.2 using mod_proxy_balancer
and mod_proxy_ajp as a replacement for our good old 1.3/mod_jk setup
as a front-end to Tomcat 5.5.
I found several problems during this process.
* redirect feature of balanced workers does not work as advertised.
The documentation for the redirect attribute states:
Redirection Route of the worker. This value is usually set
dynamically to enable safe removal of the node from the cluster. If
set all requests without session id will be redirected to the
BalancerMember that has route parametar equal as this value.
But in the source I could only found uses of this attribute in
mod_proxy_balancer.c (find_session_route) where the associated code
is:
worker = find_route_worker(balancer, *route);
if (worker !PROXY_WORKER_IS_USABLE(worker)) {
/* We have a worker that is unusable.
* It can be in error or disabled, but in case
* it has a redirection set use that redirection worker.
* This enables to safely remove the member from the
* balancer. Of course you will need a some kind of
* session replication between those two remote.
*/
if (*worker-s-redirect)
worker = find_route_worker(balancer, worker-s-redirect);
/* Check if the redirect worker is usable */
if (worker !PROXY_WORKER_IS_USABLE(worker))
worker = NULL;
}
return worker;
so basically what is being done is that if a request is part of a
session and thus has to be routed to a specific worker then, if that
worker is unsuable but has a redirection then use that worker instead
(if it is usable).
So redirection is only used when handling session bound requests which
is exactly the opposite of what the documentation said.
The goal of graceful worker shutdown is therefore not achieved as
there is no way to have session bound requests stick to their origin
worker and new requests to flow to another worker.
Having worked four years ago on the lb part of mod_jk I think a simple
redirect flag would be helpful, that is flag a worker as *redirected*
to have all requests not part of a session redirected to other
workers. This will allow for graceful worker shutdown as after having
marked a worker redirected only requests to sessions initiated on that
worker will be forwarded to it, no new sessions will be created,
therefore after a while this worker will have no traffic at all
(assuming sessions do expire...).
While investigating this problem I found another one, in the same
function (find_session_route), if a worker is diagnosed in error state
and all requests coming into Apache belong to sessions initiated in
this worker then the worker will never be retryed (see code above that
only checks for redirection).
I have other problems in stock for which I have produced patches, I
will talk about those later as they are not as critical as this one.
Anybody experienced the same problem with mod_proxy_balancer?
Regards,
Mathias.
Re: Regexp-based rewriting for mod_headers?
On Thursday 28 September 2006 18:29, Garrett Rooney wrote: On 9/28/06, Nick Kew [EMAIL PROTECTED] wrote: We have a problem with DAV + SSL hardware. It appears to be the issue described in http://svn.haxx.se/users/archive-2006-03/0549.shtml It seems to me that the ability to rewrite a request header will fix that. As a generic fix, I've patched mod_headers to support regexp-based rewriting of arbitrary headers. Please review. If people like this (or if noone objects and I find the time), I'll document it and commit to /trunk/. The patch seems reasonable, but I've never messed with mod_headers, so bear with me. It should let you do something like: Header edit header regex replace-string Exactly (and support mod_headers's optional cond var). Right? Could you provide an example so I can confirm that it's working correctly? The example that fixes the DAV problem above can be stated as RequestHeader edit Destination ^https: http: Otherwise, one can dream up mod_rewrite like cases; e.g. (modulo line wrap) Header edit Location ^http://([a-z0-9-]+)\.example\.com/ http://example.com/$1/ (not sure if that makes sense, but you get my meaning:-) -- Nick Kew
Re: mod_proxy and balancer problems
On 09/29/2006 09:53 AM, Mathias Herberts wrote: So redirection is only used when handling session bound requests which is exactly the opposite of what the documentation said. The goal of graceful worker shutdown is therefore not achieved as there is no way to have session bound requests stick to their origin worker and new requests to flow to another worker. Having worked four years ago on the lb part of mod_jk I think a simple redirect flag would be helpful, that is flag a worker as *redirected* to have all requests not part of a session redirected to other IMHO opinion that is what disabled is designed for. But I admit this does not work currently. We had a discussion a while ago here about the meaning of the diferent worker stati with not an all too clear outcome from my point of view. I tend to say that disabled means: Take requests if their routing information matches mine but no other ones. stopped means: Take no requests at all. enabled means: Take all requests. Of course above statements are subject to error checking, and lb strategies for new requests. Maybe others can give a comment. In order to get this working the following steps are missing: 1. Make disabled workers accept requests if the routing information from the client matches that of the disbaled worker. 2. Extend the manager application to put a worker into stopped / non stopped state. While investigating this problem I found another one, in the same function (find_session_route), if a worker is diagnosed in error state and all requests coming into Apache belong to sessions initiated in this worker then the worker will never be retryed (see code above that only checks for redirection). This will be fixed in 2.2.4 Regards Rüdiger
Re: svn commit: r451006 - in /httpd/httpd/trunk/modules/generators: mod_cgi.c mod_cgid.c
On Thu, Sep 28, 2006 at 08:15:44PM -, [EMAIL PROTECTED] wrote:
--- httpd/httpd/trunk/modules/generators/mod_cgi.c (original)
+++ httpd/httpd/trunk/modules/generators/mod_cgi.c Thu Sep 28 13:15:42 2006
@@ -837,6 +837,11 @@
APR_BLOCK_READ, HUGE_STRING_LEN);
if (rv != APR_SUCCESS) {
+if (rv == APR_TIMEUP) {
These should use APR_STATUS_IS_TIMEUP() rather than checking against
APR_TIMEUP directly.
joe
[Fwd: Re: [PATCH] setenvif filter]
Bringing this up. again. Adds a filter that allows mod_setenvif to act on response headers. Original Message Subject: Re: [PATCH] setenvif filter Date: Wed, 31 May 2006 17:24:33 +0200 From: Francois Pesce [EMAIL PROTECTED] Reply-To: dev@httpd.apache.org To: dev@httpd.apache.org References: [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] These patches may fix the r-content_type behaviour. Are you OK with it ? -- *Francois Pesce* 2006/5/31, Brian Akins [EMAIL PROTECTED]: Francois PESCE wrote: I've discussed about a patch for mod_setenvif 2 years ago, and have coded it at that time, it is successfully used on various host in production since. You need to handle content type specially by checking r-content_type. For some reason, just doing apr_table_get(r-headers_out, Content-type) would be null, but content_type would be set. See the patch I posted a few days ago. +1 in concept -- Brian Akins Lead Systems Engineer CNN Internet Technologies -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies mod_setenvif-2-2-x-2.patch Description: Binary data mod_setenvif-2-0-x-2.patch Description: Binary data
Re: mod_proxy and balancer problems
On Sep 29, 2006, at 3:53 AM, Mathias Herberts wrote: Hi, I am in the process of evaluating Apache 2.2 using mod_proxy_balancer and mod_proxy_ajp as a replacement for our good old 1.3/mod_jk setup as a front-end to Tomcat 5.5. I found several problems during this process. ... I have other problems in stock for which I have produced patches, I will talk about those later as they are not as critical as this one. We'll keep an eye out for your patches.
Re: Regexp-based rewriting for mod_headers?
+1 on the patch -- Brian Akins Chief Operations Engineer Turner Digital Media Technologies
Re: [Fwd: Re: [PATCH] setenvif filter]
On 9/29/06, Brian Akins [EMAIL PROTECTED] wrote: Bringing this up. again. Adds a filter that allows mod_setenvif to act on response headers. FWIW, I like it :) Have been wanting to do it myself, but have not found the time. -B
Re: svn commit: r451006 - in /httpd/httpd/trunk/modules/generators: mod_cgi.c mod_cgid.c
On 9/29/06, Joe Orton [EMAIL PROTECTED] wrote:
On Thu, Sep 28, 2006 at 08:15:44PM -, [EMAIL PROTECTED] wrote:
--- httpd/httpd/trunk/modules/generators/mod_cgi.c (original)
+++ httpd/httpd/trunk/modules/generators/mod_cgi.c Thu Sep 28 13:15:42 2006
@@ -837,6 +837,11 @@
APR_BLOCK_READ, HUGE_STRING_LEN);
if (rv != APR_SUCCESS) {
+if (rv == APR_TIMEUP) {
These should use APR_STATUS_IS_TIMEUP() rather than checking against
APR_TIMEUP directly.
Fixed in 451289.
-garrett
Re: svn commit: r451006 - in /httpd/httpd/trunk/modules/generators: mod_cgi.c mod_cgid.c
On 09/29/2006 03:49 PM, Garrett Rooney wrote:
On 9/29/06, Joe Orton [EMAIL PROTECTED] wrote:
On Thu, Sep 28, 2006 at 08:15:44PM -, [EMAIL PROTECTED] wrote:
--- httpd/httpd/trunk/modules/generators/mod_cgi.c (original)
+++ httpd/httpd/trunk/modules/generators/mod_cgi.c Thu Sep 28
13:15:42 2006
@@ -837,6 +837,11 @@
APR_BLOCK_READ, HUGE_STRING_LEN);
if (rv != APR_SUCCESS) {
+if (rv == APR_TIMEUP) {
These should use APR_STATUS_IS_TIMEUP() rather than checking against
APR_TIMEUP directly.
Fixed in 451289.
You are too fast for me :-). Just wanted to fix that.
Regards
Rüdiger
Re: Regexp-based rewriting for mod_headers?
On 9/29/06, Nick Kew [EMAIL PROTECTED] wrote: On Thursday 28 September 2006 18:29, Garrett Rooney wrote: On 9/28/06, Nick Kew [EMAIL PROTECTED] wrote: We have a problem with DAV + SSL hardware. It appears to be the issue described in http://svn.haxx.se/users/archive-2006-03/0549.shtml It seems to me that the ability to rewrite a request header will fix that. As a generic fix, I've patched mod_headers to support regexp-based rewriting of arbitrary headers. Please review. If people like this (or if noone objects and I find the time), I'll document it and commit to /trunk/. The patch seems reasonable, but I've never messed with mod_headers, so bear with me. It should let you do something like: Header edit header regex replace-string Exactly (and support mod_headers's optional cond var). Right? Could you provide an example so I can confirm that it's working correctly? The example that fixes the DAV problem above can be stated as RequestHeader edit Destination ^https: http: Otherwise, one can dream up mod_rewrite like cases; e.g. (modulo line wrap) Header edit Location ^http://([a-z0-9-]+)\.example\.com/ http://example.com/$1/ (not sure if that makes sense, but you get my meaning:-) Cool. I just tested it here, and it seems to work fine. +1 to commit from me, assuming suitable docs and so forth. My only comment about the patch itself is I'm not overly thrilled about recursively processing matches, simply because every time we recurse we allocate an entire new string to hold the entire results, which seems wasteful. But on the other hand, any other solution would probably be much more complex, and we're not talking about that many levels of recursion in any sane configureation, so hey, whatever. -garrett
Re: Regexp-based rewriting for mod_headers?
On Friday 29 September 2006 15:45, Garrett Rooney wrote: My only comment about the patch itself is I'm not overly thrilled about recursively processing matches, simply because every time we recurse we allocate an entire new string to hold the entire results, which seems wasteful. But on the other hand, any other solution would probably be much more complex, and we're not talking about that many levels of recursion in any sane configureation, so hey, whatever. -garrett I wasn't sure about that myself: a once-only match might be better. It's actually something of a cop-out for dealing with headers-fixup being called multiple times over things like mod_dir/mod_negotiation internal redirects, leading to it being applied an indeterminate number of times. The alternative is to save state there. -- Nick Kew
Re: Regexp-based rewriting for mod_headers?
On Thu, Sep 28, 2006 at 05:16:24PM +0100, Nick Kew wrote: It seems to me that the ability to rewrite a request header will fix that. As a generic fix, I've patched mod_headers to support regexp-based rewriting of arbitrary headers. +1 from the peanut gallery - I've had to do similar hackery for a customer and would much prefer to see it supported ;) vh Mads Toftum -- `Darn it, who spiked my coffee with water?!' - lwall
Re: mod_proxy and balancer problems
Joost de Heer wrote: I tend to say that disabled means: Take requests if their routing information matches mine but no other ones. stopped means: Take no requests at all. enabled means: Take all requests. Wouldn't 'standby' be a better name for the functionality you described as 'disabled'? IMO 'disabled' means 'never use, not startable', and 'stopped' means 'don't use in this state, startable via admin interface'. That aligns with my definitions... -- === Jim Jagielski [|] [EMAIL PROTECTED] [|] http://www.jaguNET.com/ If you can dodge a wrench, you can dodge a ball.
Re: mod_proxy and balancer problems
On 09/29/2006 06:28 PM, Joost de Heer wrote: I tend to say that disabled means: Take requests if their routing information matches mine but no other ones. stopped means: Take no requests at all. enabled means: Take all requests. Wouldn't 'standby' be a better name for the functionality you described as 'disabled'? IMO 'disabled' means 'never use, not startable', and 'stopped' means 'don't use in this state, startable via admin interface'. Which does not match with the current implementation: You can change between enable / disable over the admin interface but not between stopped / not stopped. I do not bother what it is called, but I think we should have names for the 3 cases I described above. And we should be able to set the status bit for 'enable/disable' 'stopped/not stopped' via the admin interface to be able to do such things like a graceful shutdown of a worker or what had been called a 'local worker' in mod_jk ages ago. Regards Rüdiger
