DeltaManager not supporting useDirtyFlag=false, suggesting safety-improved DeltaManager
Hi,
like Filip state on tomcat-user some time ago that DeltaManager doesn't
support the useDirtyFlag=false attribute.
http://www.mail-archive.com/[EMAIL PROTECTED]/msg136306.html
I didn't find any documentation so I suggest (a) better documentation of
this fact, i.e. in server.xml, and (b) throwing an exception whenever
someone tries to use useDirtyFlag=false on DeltaManager, i.e. like this
in DeltaManager.java:
public void setUseDirtyFlag(boolean useDirtyFlag) {
if(useDirtyFlag == false) {
throw new IllegalArgumentException(DeltaManager only
supports useDirtyFlag = true (yet). Maybe use
SimpleTcpReplicationManager instead.);
}
this.useDirtyFlag = useDirtyFlag;
}
Alex.
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: POST recovery in JK and JK2 HEAD
THIS DOES NOT FIX the problem with the output when tomcat already sent
data to the client (see my mail date Thu, 12 Feb 2004 23:33:34 +0100),
My two cents: when the tomcat fails after data has been sent to the
client, assume that the request is complete, i.e. return JK_TRUE
(otherwise apache will add some error messages to the reply).
Ok, if you have fix for jk AND jk2, thanks to provide them quickly so we
could include in in 2.0.4 and 1.2.6 release.
This is not as easy as I thought: I have extended ajp_get_reply() and
ajp_process_callback() to keep track if the headers have already been
sent.
The following considerations have been made:
* after data has been sent to the client don't try to recover
from a tomcat has failure
* if I return JK_FALSE with recoverable = JK_FALSE apache will
append its standard error page. This might look quite ugly, but
this way we get proper access.log entries. This might also trigger
some error handling in apache (closing the socket even if client
tried keepalive and tomcat sent a content length?)
* it's up to you to decide if mod_jk should try to recover after
when no headers have been sent (see tag DISCUSSION in code)
The best thing to do after a tomcat failure (after data has been already
sent to the client) would be to mark it as not recoverable, write a proper
log entry in access.log and to force apache to close this connection
immediately without sending any other data. This should handle any
problems arising from chunked responses and/or content-length headers. But
I don't know how to implement that.
The included patch is only for jk, not jk2.
Alex.
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
? jtc.20040208.zip
? jtc.20040215.zip
? jk/native/autom4te.cache
? jk/native/out
? jk/native/apache-1.3/mod_jk.so.0.0.0
? jk/native/jni/Makefile
? jk/native/scripts/build/unix/config.guess
? jk/native/scripts/build/unix/config.sub
? jk/native/scripts/build/unix/install-sh
? jk/native/scripts/build/unix/ltmain.sh
? jk/native/scripts/build/unix/missing
? jk/native/scripts/build/unix/mkinstalldirs
Index: jk/native/common/jk_ajp_common.c
===
RCS file: /home/cvspublic/jakarta-tomcat-connectors/jk/native/common/jk_ajp_common.c,v
retrieving revision 1.47
diff -u -b -r1.47 jk_ajp_common.c
--- jk/native/common/jk_ajp_common.c11 Feb 2004 09:49:49 - 1.47
+++ jk/native/common/jk_ajp_common.c16 Feb 2004 11:49:35 -
@@ -1174,7 +1174,8 @@
(const char * const *)res.header_values,
res.num_headers);
}
-break;
+// break;
+ return JK_AJP13_SEND_HEADERS;
case JK_AJP13_SEND_BODY_CHUNK:
{
@@ -1261,6 +1262,7 @@
ajp_endpoint_t *p,
ajp_operation_t *op)
{
+int headeratclient = JK_FALSE;
/* Start read all reply message */
while(1) {
int rc = 0;
@@ -1279,11 +1281,40 @@
}
if(!ajp_connection_tcp_get_message(p, op-reply, l)) {
+/* we just can't recover, unset recover flag */
+ if(JK_FALSE == headeratclient) {
jk_log(l, JK_LOG_ERROR,
Error reading reply from tomcat.
- Tomcat is down or network problems.\n);
-/* we just can't recover, unset recover flag */
+ Tomcat is down or network problems.
+ No response has been sent to the client (yet)\n);
+ /* communication with tomcat has been interrupted BEFORE
+* headers have been sent to the client. */
+ /* DISCUSSION: As we suppose that tomcat has already started
+* to process the query we think it's unrecoverable (and we
+* should not retry or switch to another tomcat in the
+* cluster). */
+ op-recoverable = JK_FALSE;
+ /* we want to display the webservers error page, therefore
+* we return JK_FALSE */
return JK_FALSE;
+ } else {
+jk_log(l, JK_LOG_ERROR,
+ Error reading reply from tomcat.
+ Tomcat is down or network problems.
+ Part of the response has already been sent to the client\n);
+ /* communication with tomcat has been interrupted AFTER
+* headers have been sent to the client. */
+ /* headers (and maybe parts of the body) have already been
+* sent, therefore the response is complete in a sense
+* that nobody should append any data, especially no 500 error
+* page of the webserver! */
+ /* BUT if you retrun JK_TRUE you have a 200 (OK) code in your
+* in your apache access.log instead
Re: POST recovery in JK and JK2 HEAD
I commited the fixes for the POST recovery in JK and JK2 when in LB mode which should solve this major problem. Could Alexander who is the original reporter validate the fix ? YES and NO: Post recovery works now in the scenario, the loadbalancer doesn't forget the post body when a tomcat fails. THIS DOES NOT FIX the problem with the output when tomcat already sent data to the client (see my mail date Thu, 12 Feb 2004 23:33:34 +0100), but you have already been discussing this under the subject Mod_Jk2 - Default Worker. My two cents: when the tomcat fails after data has been sent to the client, assume that the request is complete, i.e. return JK_TRUE (otherwise apache will add some error messages to the reply). Alex. -- Alexander Schwartz ([EMAIL PROTECTED]) http://www.ahus1.de - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Am Mi, den 11.02.2004 schrieb Henri Gomez um 10:57:
Ok, I commited changes in CVS which should fix problem in POST recovery.
Thanks to take a look at it
OK, POST recovery seems to work now in the scenario i have described:
Tomcat fails after receiving the request but before sending a reply.
POST data is recovered and resent to the other tomcat. Good!
I now have the trace I promised for the second challenge: tomcat fails
while sending data to the client. Here comes the test:
* set up two tomcats
* use the extended EchoFilter.java that sends one byte
each second for ten seconds
* send a request to the first one (take the login.html provided)
* use kill -9 [pid] on the first one
* see the following reply appearing in the window of your browser:
-
...HTTP/1.1 200 OK
Date: Thu, 12 Feb 2004 22:15:30 GMT
Server: Apache/1.3.29 (Debian GNU/Linux) PHP/4.3.3 mod_ssl/2.8.14 OpenSSL/0.9.7b
mod_jk/1.2.6-dev
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
Keep-Alive: timeout=15, max=99
Connection: Keep-Alive, Keep-Alive
Transfer-Encoding: chunked, chunked
Content-Type: text/plain
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
..login=pin=
-
The first tomcat has been killed after three seconds (browser has
received the header and shows one dot per second). Then another header
and another body is sent to the browser and displayed -- the browser
accepts the second header as part of the body.
If you would have used telnet to trigger the request you would have seen
two Headers.
This is not specific to POST-Requests, the same is true for
GET-Requests. Do you still want to recover in this scenario?
Alex.
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
package com.direkt1822.joba.filter;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* This is a filter that sends all data sent from the client back to the client
* @author Alexander Schwartz (1822direkt) 2004
* @version $Id: EchoFilter.java,v 1.1 2004/01/22 12:10:39 ahus1 Exp $
*/
public class EchoFilter implements Filter {
public void init(FilterConfig filterConfig) throws ServletException {
}
public void doFilter(
ServletRequest request,
ServletResponse response,
FilterChain chain)
throws IOException, ServletException {
final HttpServletRequest httpRequest =
(HttpServletRequest)request;
final HttpServletResponse httpResponse =
(HttpServletResponse)response;
BufferedReader reader = httpRequest.getReader();
PrintWriter writer = httpResponse.getWriter();
String line;
try {
for(int i=0; i10; ++i) {
Thread.sleep(1000);
writer.print(.);
writer.flush();
} } catch (InterruptedException e) {
}
while((line = reader.readLine())!=null) {
writer.println(line);
}
}
public void destroy() {
}
}
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (172)]: Into
jk_uri_worker_map_t::uri_worker_map_alloc
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (375)]: Into
jk_uri_worker_map_t::uri_worker_map_open
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (396)]:
jk_uri_worker_map_t::uri_worker_map_open, rule map size is 1
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (321)]: Into
jk_uri_worker_map_t::uri_worker_map_open, match rule /JOBa1822Web/=lb was added
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (408)]: Into
jk_uri_worker_map_t::uri_worker_map_open, there are 1 rules
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (422)]:
jk_uri_worker_map_t::uri_worker_map_open, done
[Thu Feb 12 23:25:18 2004] [jk_worker.c (88)]: Into wc_open
[Thu Feb 12 23:25:18 2004] [jk_worker.c (222)]: Into build_worker_map, creating 3
workers
[Thu Feb 12 23:25:18 2004] [jk_worker.c (228)]: build_worker_map, creating worker tc1
[Thu Feb 12 23:25:18 2004] [jk_worker.c (148)]: Into wc_create_worker
[Thu Feb 12 23:25:18 2004] [jk_worker.c (162)]: wc_create_worker, about to create
instance tc1 of ajp13
[Thu Feb 12 23:25:18 2004] [jk_ajp13_worker.c (108)]: Into ajp13_worker_factory
[Thu Feb 12 23:25:18 2004] [jk_worker.c (171)]: wc_create_worker, about to validate
and init tc1
[Thu Feb 12 23:25:18 2004] [jk_ajp_common.c (1488)]: Into jk_worker_t::validate
[Thu Feb 12 23:25:18 2004] [jk_ajp_common.c (1509)]: In jk_worker_t::validate for
worker tc1 contact is localhost:9010
[Thu Feb 12 23:25:18 2004] [jk_ajp_common.c (1542)]: Into jk_worker_t::init
[Thu Feb 12 23:25:18 2004
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Am Di, den 10.02.2004 schrieb Henri Gomez um 15:52:
To fix this serious bug we should modify the current service API.
We should store the POSTED datas in the jk_ws_service_t area and
not in ajp_operation_t, as such the first worker will feed the POST
datas in an area which will be available to the second one.
Aternate idea are of course more than welcome;
As I understand it mod_jk is stable, almost deprecated, and mod_jk2
is the future. Therefore I suggest:
* no API change in mod_jk
* mark the operation as unrecoverable when an error occurs after
the headers have been sent to the tomcat, that is only fixing
bugs and trying not to do big changes
* concentrate your efforts on mod_jk2 and give people a reason to
upgrade :)
Marking the operation as unrecoverable might also solve the other
(lb-)problem: when you shut down the first tomcat after it has already
sent some data to the client mod_jk forwards the request to the second
tomcat and appends the response of the second tomcat to the response of
the first tomcat. I'll deliver a sample jk.log soon.
I have prepared a patch for 1.2.4/1.2.5, and I am happy to port it to
HEAD -- if someone checks that it leaves no loose ends.
Thanks for taking this serious.
Alex.
PS: this empty POST bug is not the explanation for the session mixup
(yet)? Any ideas on that one?
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
--- jk_ajp_common.c.old Fri Jan 23 16:26:12 2004
+++ jk_ajp_common.c Fri Jan 23 16:28:02 2004
@@ -963,10 +963,16 @@
* or Tomcat crashed. In any case we cannot recover this.
*/
+// as+gj/20040123: we don't want to try the buggy POST recovery as it
+// seems to repost buggy data
+// BUT: doesn't work!
+// op-recoverable = JK_FALSE;
+
jk_log(l, JK_LOG_DEBUG,
ajp_send_request 2:
- request body to send %d - request body to resend %d\n,
- ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN);
+ request body to send %d - request body to resend %d/%d\n,
+ ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN,
+ jk_b_get_len(op-post) - AJP_HEADER_LEN);
/*
* POST recovery job is done here.
@@ -977,10 +983,18 @@
* remote Tomcat
*/
if (jk_b_get_len(op-post) AJP_HEADER_LEN) {
-if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
-jk_log(l, JK_LOG_ERROR, Error resending request body\n);
-return JK_FALSE;
-}
+ // as+gj/20040123: seems to be buggy, doesn't fit the debug output
+ // above, customers complain about seeing wrong user's data when doing
+ // a login-POST
+ // if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
+ // jk_log(l, JK_LOG_ERROR, Error resending request body\n);
+ // return JK_FALSE;
+ // }
+
+ // as+gj/20040123 recovery is disabled above shoudn't happen!
+ jk_log(l, JK_LOG_ERROR, Shouldn't be here resending request body! (as+gj/20040123)\n);
+ op-recoverable = JK_FALSE;
+ return JK_FALSE;
}
else {
/* We never sent any POST data and we check if we have to send at
@@ -1009,6 +1023,8 @@
s-content_read = len;
if (!ajp_connection_tcp_send_message(ae, op-post, l)) {
jk_log(l, JK_LOG_ERROR, Error sending request body\n);
+// as+gj/20040123
+op-recoverable = JK_FALSE;
return JK_FALSE;
}
}
@@ -1279,6 +1295,9 @@
* receiving upload data and we must consider that
* operation is no more recoverable
*/
+// as+gj/20040123:
+// don't want to recover
+op-recoverable = JK_FALSE;
if (! op-recoverable) {
*is_recoverable_error = JK_FALSE;
jk_log(l, JK_LOG_ERROR,
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Hello Henri, Am Fr, den 06.02.2004 schrieb Henri Gomez um 10:15: Strange, also I need a complete mod_jk.log for study. OK, here is the first part of my tests with mod_jk from CVS: Test 1: * run apache/mod_jk with cluster and failover (config included) * tc1 is offline * tc2 is a real tomcat After the apache start the first request (post of login.html, included) is directed to tc1, but it doesn't response. The request is directed to tc2, and EchoFilter.java echos the POST-data to the client. This is OK! Trace: jk.log.onlytc2runs Test 2: * run apache/mod_jk with cluster and failover (config included) * tc1 is a simple netcat (nc -l -p 9010 localhost) * tc2 is a real tomcat After the apache starts the first request (post of login.html, included) is directed to tc1. netcat answers the connection and prints out some garbage on the console. Then you kill netcat (^C). Then mod_jk redirects the request to tc2. == here mod_jk forgets the POST-data and no data is shown to the client! Trace: jk.log.netcatdiedfailovertotc2 Tomorrow I'll trace the problem when tc1 fails after it has already sent header and data to the client. Alex. -- Alexander Schwartz ([EMAIL PROTECTED]) http://www.ahus1.de [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (172)]: Into jk_uri_worker_map_t::uri_worker_map_alloc [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (375)]: Into jk_uri_worker_map_t::uri_worker_map_open [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (396)]: jk_uri_worker_map_t::uri_worker_map_open, rule map size is 1 [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (321)]: Into jk_uri_worker_map_t::uri_worker_map_open, match rule /JOBa1822Web/=lb was added [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (408)]: Into jk_uri_worker_map_t::uri_worker_map_open, there are 1 rules [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (422)]: jk_uri_worker_map_t::uri_worker_map_open, done [Mon Feb 09 22:36:01 2004] [jk_worker.c (88)]: Into wc_open [Mon Feb 09 22:36:01 2004] [jk_worker.c (222)]: Into build_worker_map, creating 3 workers [Mon Feb 09 22:36:01 2004] [jk_worker.c (228)]: build_worker_map, creating worker tc1 [Mon Feb 09 22:36:01 2004] [jk_worker.c (148)]: Into wc_create_worker [Mon Feb 09 22:36:01 2004] [jk_worker.c (162)]: wc_create_worker, about to create instance tc1 of ajp13 [Mon Feb 09 22:36:01 2004] [jk_ajp13_worker.c (108)]: Into ajp13_worker_factory [Mon Feb 09 22:36:01 2004] [jk_worker.c (171)]: wc_create_worker, about to validate and init tc1 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1468)]: Into jk_worker_t::validate [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1489)]: In jk_worker_t::validate for worker tc1 contact is localhost:9010 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1522)]: Into jk_worker_t::init [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1542)]: In jk_worker_t::init, setting socket timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1549)]: In jk_worker_t::init, setting socket keepalive to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1556)]: In jk_worker_t::init, setting cache timeout to 15 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1563)]: In jk_worker_t::init, setting connect timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1570)]: In jk_worker_t::init, setting reply timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1577)]: In jk_worker_t::init, setting prepost timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_worker.c (187)]: wc_create_worker, done [Mon Feb 09 22:36:01 2004] [jk_worker.c (238)]: build_worker_map, removing old tc1 worker [Mon Feb 09 22:36:01 2004] [jk_worker.c (228)]: build_worker_map, creating worker tc2 [Mon Feb 09 22:36:01 2004] [jk_worker.c (148)]: Into wc_create_worker [Mon Feb 09 22:36:01 2004] [jk_worker.c (162)]: wc_create_worker, about to create instance tc2 of ajp13 [Mon Feb 09 22:36:01 2004] [jk_ajp13_worker.c (108)]: Into ajp13_worker_factory [Mon Feb 09 22:36:01 2004] [jk_worker.c (171)]: wc_create_worker, about to validate and init tc2 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1468)]: Into jk_worker_t::validate [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1489)]: In jk_worker_t::validate for worker tc2 contact is localhost:9009 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1522)]: Into jk_worker_t::init [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1542)]: In jk_worker_t::init, setting socket timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1549)]: In jk_worker_t::init, setting socket keepalive to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1556)]: In jk_worker_t::init, setting cache timeout to 15 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1563)]: In jk_worker_t::init, setting connect timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1570)]: In jk_worker_t::init, setting reply timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1577)]: In jk_worker_t::init, setting prepost timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_worker.c (187)]: wc_create_worker
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Am Fr, den 06.02.2004 schrieb Henri Gomez um 10:13: Could you try with the latest code from CVS, since your patch seems to be for an older version ? I've tested with mod_jk 1.2.4/1.2.5, but I will do it again with the latest Version from CVS with jk.log in DEBUG mode. Will be first thing Monday morning. I'd like to see test case and dumps. What is a dump? What are your requirements for a good test case? I also wonder why the tomcat failed since using the prepost_timeout ping/pong could help you there. It would help, but it has not been released yet (mod_jk 1.2.5 is latest released version, but it will be in mod_jk 1.2.6) -- but it doesn't solve problems when a tomcat fails while processing the request. netcat should handle AJP13, not HTTP, so it should be 8009. Sure, yes, it was listening on 8009 and was receiving AJP13. Sorry. Questions : - Did Tomcat failed while handling the request or was allready broken before ? - Did you clients used chunked transfert ? - You say that sometimes after a POST-failover body data from an old request is sent to the other tomcat when the first connection failed. Do you means that the other tomcat get POST DATA which was never sent by the browser in this request ? Strange since the op-post is reset each time we service a new request. As far as I was able to trace when the session mixup happend tomcat was processing data that has been sent about 15 minutes earlier and delivered the result to another user. I think this was due to the fact that mod_jk mixed up the POST data. I don't know about chunked transfers, but this is unlikely as the data was a normal form with data less than 100 bytes. What I was able to test was empty POST after the first tomcat failed while processing the request and mod_jk retried on another tomcat. Wait for my jk.logs. Alex. -- Alexander Schwartz ([EMAIL PROTECTED]) http://www.ahus1.de - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
I'd really like to use mod_jk open source software in our production
environment, but I need to make sure it's strong enough for production.
I need some advice on the two issues (as far as I can see there has been
no commit on CVS and/or discussion results)
* do you think the POST-recovery caused the session mix-up?
* does our patch successfully eliminate the problem by disabling
POST-recovery?
I'd like to see this recovery fixed instead of removing it.
but I need something working now :( -- thefore I asked if my code is a
working, although i know it is not perfect seen with your eyes.
The idea is to use buffers to store request datas (headers/datas).
OK, understood.
What's the exact problem.
You send request (headers / datas) to tomcat1.
This one fail when sending reply or didn't reply at all ?
I don't know when the session mix up happens, and I was unable to
reproduce it. But I was to reproduce the following:
(a) mod_jk sends the request and data to tomcat1. Tomcat1 closes the
connection. mod_jk tries to recover and sends the request with no data
to the second tomcat. Sending no data is definitely wrong. I also argue
that this is dangerous as a transaction may be triggered twice by
sending the request a second time.
Use netcat (nc -l -p 8080) to simulate a tomcat1 on port 8080
receiving the headers. Press ^C to simulate the failure of tomcat1. Use
a real tomcat2 for failover.
(b) mod_jk sends the request and data to tomcat1. mod_jk receives
header and data from tomcat, but the response is not yet complete.
tomcat1 closes the connection. mod_jk tries to recover and sends the
request with no data to the second tomcat. The output of the second
tomcat is appended to the output of the first tomcat. This is definitely
wrong.
I used the attached filter to set up a test case for this. Use two
tomcats, and shut down the first tomcat when it is processing its
request.
Alex.
--- jk_ajp_common.c.old Fri Jan 23 16:26:12 2004
+++ jk_ajp_common.c Fri Jan 23 16:28:02 2004
@@ -963,10 +963,16 @@
* or Tomcat crashed. In any case we cannot recover this.
*/
+// as+gj/20040123: we don't want to try the buggy POST recovery as it
+// seems to repost buggy data
+// BUT: doesn't work!
+// op-recoverable = JK_FALSE;
+
jk_log(l, JK_LOG_DEBUG,
ajp_send_request 2:
- request body to send %d - request body to resend %d\n,
- ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN);
+ request body to send %d - request body to resend %d/%d\n,
+ ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN,
+ jk_b_get_len(op-post) - AJP_HEADER_LEN);
/*
* POST recovery job is done here.
@@ -977,10 +983,18 @@
* remote Tomcat
*/
if (jk_b_get_len(op-post) AJP_HEADER_LEN) {
-if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
-jk_log(l, JK_LOG_ERROR, Error resending request body\n);
-return JK_FALSE;
-}
+ // as+gj/20040123: seems to be buggy, doesn't fit the debug output
+ // above, customers complain about seeing wrong user's data when doing
+ // a login-POST
+ // if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
+ // jk_log(l, JK_LOG_ERROR, Error resending request body\n);
+ // return JK_FALSE;
+ // }
+
+ // as+gj/20040123 recovery is disabled above shoudn't happen!
+ jk_log(l, JK_LOG_ERROR, Shouldn't be here resending request body!
(as+gj/20040123)\n);
+ op-recoverable = JK_FALSE;
+ return JK_FALSE;
}
else {
/* We never sent any POST data and we check if we have to send at
@@ -1009,6 +1023,8 @@
s-content_read = len;
if (!ajp_connection_tcp_send_message(ae, op-post, l)) {
jk_log(l, JK_LOG_ERROR, Error sending request body\n);
+// as+gj/20040123
+op-recoverable = JK_FALSE;
return JK_FALSE;
}
}
@@ -1279,6 +1295,9 @@
* receiving upload data and we must consider that
* operation is no more recoverable
*/
+// as+gj/20040123:
+// don't want to recover
+op-recoverable = JK_FALSE;
if (! op-recoverable) {
*is_recoverable_error = JK_FALSE;
jk_log(l, JK_LOG_ERROR,
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
package com.direkt1822.joba.filter;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.Filter;
import
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Hello Henri, hello mod_jk-Developers,
Am Di, den 27.01.2004 schrieb Henri Gomez um 18:00:
we are currently experiencing some problems with mod_jk with
loadbalancing:
(A) when a post-request fails while receiving data from tomcat, the
loadbalancer tries to send the request to the other tomcat, but
forgets the post body content (i.e. login and password submitted by
the user). The size of the body content is only a few bytes ( 100), so
it's not the known problem with bodies longer than 8k.
(B) it seems that sometimes after a POST-failover body data from an
old request is sent to the other tomcat when the first connection
failed. This leads to a session mix-up user mix-up!
[...]
It used to works but it was before the content-header add-on code.
This piece of is a nightmare, and I see no easier solution than copying
the HEAD+POST in temp buffer but I'd like to have Bill, Mladen and JFC
advices...
I'd really like to use mod_jk open source software in our production
environment, but I need to make sure it's strong enough for production.
I need some advice on the two issues (as far as I can see there has been
no commit on CVS and/or discussion results)
* do you think the POST-recovery caused the session mix-up?
* does our patch successfully eliminate the problem by disabling
POST-recovery?
The patch is attached.
Thanks a lot,
Alexander.
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
--- jk_ajp_common.c.old Fri Jan 23 16:26:12 2004
+++ jk_ajp_common.c Fri Jan 23 16:28:02 2004
@@ -963,10 +963,16 @@
* or Tomcat crashed. In any case we cannot recover this.
*/
+// as+gj/20040123: we don't want to try the buggy POST recovery as it
+// seems to repost buggy data
+// BUT: doesn't work!
+// op-recoverable = JK_FALSE;
+
jk_log(l, JK_LOG_DEBUG,
ajp_send_request 2:
- request body to send %d - request body to resend %d\n,
- ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN);
+ request body to send %d - request body to resend %d/%d\n,
+ ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN,
+ jk_b_get_len(op-post) - AJP_HEADER_LEN);
/*
* POST recovery job is done here.
@@ -977,10 +983,18 @@
* remote Tomcat
*/
if (jk_b_get_len(op-post) AJP_HEADER_LEN) {
-if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
-jk_log(l, JK_LOG_ERROR, Error resending request body\n);
-return JK_FALSE;
-}
+ // as+gj/20040123: seems to be buggy, doesn't fit the debug output
+ // above, customers complain about seeing wrong user's data when doing
+ // a login-POST
+ // if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
+ // jk_log(l, JK_LOG_ERROR, Error resending request body\n);
+ // return JK_FALSE;
+ // }
+
+ // as+gj/20040123 recovery is disabled above shoudn't happen!
+ jk_log(l, JK_LOG_ERROR, Shouldn't be here resending request body! (as+gj/20040123)\n);
+ op-recoverable = JK_FALSE;
+ return JK_FALSE;
}
else {
/* We never sent any POST data and we check if we have to send at
@@ -1009,6 +1023,8 @@
s-content_read = len;
if (!ajp_connection_tcp_send_message(ae, op-post, l)) {
jk_log(l, JK_LOG_ERROR, Error sending request body\n);
+// as+gj/20040123
+op-recoverable = JK_FALSE;
return JK_FALSE;
}
}
@@ -1279,6 +1295,9 @@
* receiving upload data and we must consider that
* operation is no more recoverable
*/
+// as+gj/20040123:
+// don't want to recover
+op-recoverable = JK_FALSE;
if (! op-recoverable) {
*is_recoverable_error = JK_FALSE;
jk_log(l, JK_LOG_ERROR,
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors: jk_ajp_common.c
Am Di, den 27.01.2004 schrieb Henri Gomez um 18:00: It used to works but it was before the content-header add-on code. This piece of is a nightmare, and I see no easier solution than copying the HEAD+POST in temp buffer but I'd like to have Bill, Mladen and JFC advices... My code tries to avoid post recovery completly as my application is at the moment not safe to process a (POST) reqest twice. I wonder if post recovery is really needed. The most common error for post recovery is the We will know only at read time if the remote host closed the connection (half-closed state - FIN-WAIT2) (stated i.e. at ajp_get_reply() function comment) With ping/pong avaliable, is it still needed? As I looked deeper into you code I found that mod_jk retries the request even after headers and/or parts of the body have already been sent to the client. To me it seems that ajp_get_reply() needs some extended return value after it has called ajp_process_callback() to process JK_AJP13_SEND_HEADERS or JK_AJP13_SEND_BODY_CHUNK to set op-recoverable to JK_FALSE. Again: my code avoids this complicated decisions by promoting less recovery mechanisms, but early and clean failure :) Cheers, Alex. -- Alexander Schwartz ([EMAIL PROTECTED]) http://www.ahus1.de - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
