Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Am Mi, den 11.02.2004 schrieb Henri Gomez um 10:57:
Ok, I commited changes in CVS which should fix problem in POST recovery.
Thanks to take a look at it
OK, POST recovery seems to work now in the scenario i have described:
Tomcat fails after receiving the request but before sending a reply.
POST data is recovered and resent to the other tomcat. Good!
I now have the trace I promised for the second challenge: tomcat fails
while sending data to the client. Here comes the test:
* set up two tomcats
* use the extended EchoFilter.java that sends one byte
each second for ten seconds
* send a request to the first one (take the login.html provided)
* use kill -9 [pid] on the first one
* see the following reply appearing in the window of your browser:
-
...HTTP/1.1 200 OK
Date: Thu, 12 Feb 2004 22:15:30 GMT
Server: Apache/1.3.29 (Debian GNU/Linux) PHP/4.3.3 mod_ssl/2.8.14 OpenSSL/0.9.7b
mod_jk/1.2.6-dev
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
Keep-Alive: timeout=15, max=99
Connection: Keep-Alive, Keep-Alive
Transfer-Encoding: chunked, chunked
Content-Type: text/plain
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache
..login=pin=
-
The first tomcat has been killed after three seconds (browser has
received the header and shows one dot per second). Then another header
and another body is sent to the browser and displayed -- the browser
accepts the second header as part of the body.
If you would have used telnet to trigger the request you would have seen
two Headers.
This is not specific to POST-Requests, the same is true for
GET-Requests. Do you still want to recover in this scenario?
Alex.
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
package com.direkt1822.joba.filter;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
/**
* This is a filter that sends all data sent from the client back to the client
* @author Alexander Schwartz (1822direkt) 2004
* @version $Id: EchoFilter.java,v 1.1 2004/01/22 12:10:39 ahus1 Exp $
*/
public class EchoFilter implements Filter {
public void init(FilterConfig filterConfig) throws ServletException {
}
public void doFilter(
ServletRequest request,
ServletResponse response,
FilterChain chain)
throws IOException, ServletException {
final HttpServletRequest httpRequest =
(HttpServletRequest)request;
final HttpServletResponse httpResponse =
(HttpServletResponse)response;
BufferedReader reader = httpRequest.getReader();
PrintWriter writer = httpResponse.getWriter();
String line;
try {
for(int i=0; i10; ++i) {
Thread.sleep(1000);
writer.print(.);
writer.flush();
} } catch (InterruptedException e) {
}
while((line = reader.readLine())!=null) {
writer.println(line);
}
}
public void destroy() {
}
}
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (172)]: Into
jk_uri_worker_map_t::uri_worker_map_alloc
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (375)]: Into
jk_uri_worker_map_t::uri_worker_map_open
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (396)]:
jk_uri_worker_map_t::uri_worker_map_open, rule map size is 1
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (321)]: Into
jk_uri_worker_map_t::uri_worker_map_open, match rule /JOBa1822Web/=lb was added
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (408)]: Into
jk_uri_worker_map_t::uri_worker_map_open, there are 1 rules
[Thu Feb 12 23:25:18 2004] [jk_uri_worker_map.c (422)]:
jk_uri_worker_map_t::uri_worker_map_open, done
[Thu Feb 12 23:25:18 2004] [jk_worker.c (88)]: Into wc_open
[Thu Feb 12 23:25:18 2004] [jk_worker.c (222)]: Into build_worker_map, creating 3
workers
[Thu Feb 12 23:25:18 2004] [jk_worker.c (228)]: build_worker_map, creating worker tc1
[Thu Feb 12 23:25:18 2004] [jk_worker.c (148)]: Into wc_create_worker
[Thu Feb 12 23:25:18 2004] [jk_worker.c (162)]: wc_create_worker, about to create
instance tc1 of ajp13
[Thu Feb 12 23:25:18 2004] [jk_ajp13_worker.c (108)]: Into ajp13_worker_factory
[Thu Feb 12 23:25:18 2004] [jk_worker.c (171)]: wc_create_worker, about to validate
and init tc1
[Thu Feb 12 23:25:18 2004] [jk_ajp_common.c (1488)]: Into jk_worker_t::validate
[Thu Feb 12 23:25:18 2004] [jk_ajp_common.c (1509)]: In jk_worker_t::validate for
worker tc1 contact is localhost:9010
[Thu Feb 12 23:25:18 2004] [jk_ajp_common.c (1542)]: Into jk_worker_t::init
[Thu Feb 12 23:25:18 2004]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Alexander Schwartz a écrit : Am Di, den 10.02.2004 schrieb Henri Gomez um 15:52: To fix this serious bug we should modify the current service API. We should store the POSTED datas in the jk_ws_service_t area and not in ajp_operation_t, as such the first worker will feed the POST datas in an area which will be available to the second one. Aternate idea are of course more than welcome; As I understand it mod_jk is stable, almost deprecated, and mod_jk2 is the future. Therefore I suggest: * no API change in mod_jk * mark the operation as unrecoverable when an error occurs after the headers have been sent to the tomcat, that is only fixing bugs and trying not to do big changes * concentrate your efforts on mod_jk2 and give people a reason to upgrade :) Marking the operation as unrecoverable might also solve the other (lb-)problem: when you shut down the first tomcat after it has already sent some data to the client mod_jk forwards the request to the second tomcat and appends the response of the second tomcat to the response of the first tomcat. I'll deliver a sample jk.log soon. I have prepared a patch for 1.2.4/1.2.5, and I am happy to port it to HEAD -- if someone checks that it leaves no loose ends. Thanks for taking this serious. Alex. PS: this empty POST bug is not the explanation for the session mixup (yet)? Any ideas on that one? Well I'm sure we could update the API to have this problem fixed. The post buffer used for recovery should be create in service and passed to the differents workers, making the POST recovery working for ajp and lb workers. If nobody have objections I'll take a look at this since I don't like the idea of having something working for only HALF of HTTP protocol. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Alexander Schwartz a écrit : Am Di, den 10.02.2004 schrieb Henri Gomez um 15:52: To fix this serious bug we should modify the current service API. We should store the POSTED datas in the jk_ws_service_t area and not in ajp_operation_t, as such the first worker will feed the POST datas in an area which will be available to the second one. Aternate idea are of course more than welcome; As I understand it mod_jk is stable, almost deprecated, and mod_jk2 is the future. Therefore I suggest: * no API change in mod_jk * mark the operation as unrecoverable when an error occurs after the headers have been sent to the tomcat, that is only fixing bugs and trying not to do big changes * concentrate your efforts on mod_jk2 and give people a reason to upgrade :) Marking the operation as unrecoverable might also solve the other (lb-)problem: when you shut down the first tomcat after it has already sent some data to the client mod_jk forwards the request to the second tomcat and appends the response of the second tomcat to the response of the first tomcat. I'll deliver a sample jk.log soon. Ok, I commited changes in CVS which should fix problem in POST recovery. Thanks to take a look at it - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Alexander Schwartz a écrit : Hello Henri, Am Fr, den 06.02.2004 schrieb Henri Gomez um 10:15: Strange, also I need a complete mod_jk.log for study. OK, here is the first part of my tests with mod_jk from CVS: Test 1: * run apache/mod_jk with cluster and failover (config included) * tc1 is offline * tc2 is a real tomcat After the apache start the first request (post of login.html, included) is directed to tc1, but it doesn't response. The request is directed to tc2, and EchoFilter.java echos the POST-data to the client. This is OK! Trace: jk.log.onlytc2runs Test 2: * run apache/mod_jk with cluster and failover (config included) * tc1 is a simple netcat (nc -l -p 9010 localhost) * tc2 is a real tomcat After the apache starts the first request (post of login.html, included) is directed to tc1. netcat answers the connection and prints out some garbage on the console. Then you kill netcat (^C). Then mod_jk redirects the request to tc2. == here mod_jk forgets the POST-data and no data is shown to the client! Well netcat didn't handle ajp13 so it may be the problem. Strange, jk see it should send the 19 bytes to the second tomcat but didn't send the exact code. Could you send also the login.html ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Alexander Schwartz a écrit : Hello Henri, Am Fr, den 06.02.2004 schrieb Henri Gomez um 10:15: Strange, also I need a complete mod_jk.log for study. OK, here is the first part of my tests with mod_jk from CVS: Test 1: * run apache/mod_jk with cluster and failover (config included) * tc1 is offline * tc2 is a real tomcat After the apache start the first request (post of login.html, included) is directed to tc1, but it doesn't response. The request is directed to tc2, and EchoFilter.java echos the POST-data to the client. This is OK! Trace: jk.log.onlytc2runs Test 2: * run apache/mod_jk with cluster and failover (config included) * tc1 is a simple netcat (nc -l -p 9010 localhost) * tc2 is a real tomcat After the apache starts the first request (post of login.html, included) is directed to tc1. netcat answers the connection and prints out some garbage on the console. Then you kill netcat (^C). Then mod_jk redirects the request to tc2. == here mod_jk forgets the POST-data and no data is shown to the client! Trace: jk.log.netcatdiedfailovertotc2 Ok, I've got the same settings here, I'll investigate and try to fix :) - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Henri Gomez a écrit : Alexander Schwartz a écrit : Hello Henri, Am Fr, den 06.02.2004 schrieb Henri Gomez um 10:15: Strange, also I need a complete mod_jk.log for study. OK, here is the first part of my tests with mod_jk from CVS: Test 1: * run apache/mod_jk with cluster and failover (config included) * tc1 is offline* tc2 is a real tomcat After the apache start the first request (post of login.html, included) is directed to tc1, but it doesn't response. The request is directed to tc2, and EchoFilter.java echos the POST-data to the client. This is OK! Trace: jk.log.onlytc2runs Test 2: * run apache/mod_jk with cluster and failover (config included) * tc1 is a simple netcat (nc -l -p 9010 localhost)* tc2 is a real tomcat After the apache starts the first request (post of login.html, included) is directed to tc1. netcat answers the connection and prints out some garbage on the console. Then you kill netcat (^C). Then mod_jk redirects the request to tc2. == here mod_jk forgets the POST-data and no data is shown to the client! Trace: jk.log.netcatdiedfailovertotc2 Ok, I've got the same settings here, I'll investigate and try to fix :) I see where is the problem. The POST is saved in op-post by ajp service, but in lb mode, if the first ajp connection failed, lb try to serv the second one and sus recall ajp service WHICH reset the op-post ;( I should now see how to fix the problem (not easy). - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
I see where is the problem. The POST is saved in op-post by ajp service, but in lb mode, if the first ajp connection failed, lb try to serv the second one and sus recall ajp service WHICH reset the op-post ;( I should now see how to fix the problem (not easy). To fix this serious bug we should modify the current service API. We should store the POSTED datas in the jk_ws_service_t area and not in ajp_operation_t, as such the first worker will feed the POST datas in an area which will be available to the second one. Aternate idea are of course more than welcome; - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Am Di, den 10.02.2004 schrieb Henri Gomez um 15:52:
To fix this serious bug we should modify the current service API.
We should store the POSTED datas in the jk_ws_service_t area and
not in ajp_operation_t, as such the first worker will feed the POST
datas in an area which will be available to the second one.
Aternate idea are of course more than welcome;
As I understand it mod_jk is stable, almost deprecated, and mod_jk2
is the future. Therefore I suggest:
* no API change in mod_jk
* mark the operation as unrecoverable when an error occurs after
the headers have been sent to the tomcat, that is only fixing
bugs and trying not to do big changes
* concentrate your efforts on mod_jk2 and give people a reason to
upgrade :)
Marking the operation as unrecoverable might also solve the other
(lb-)problem: when you shut down the first tomcat after it has already
sent some data to the client mod_jk forwards the request to the second
tomcat and appends the response of the second tomcat to the response of
the first tomcat. I'll deliver a sample jk.log soon.
I have prepared a patch for 1.2.4/1.2.5, and I am happy to port it to
HEAD -- if someone checks that it leaves no loose ends.
Thanks for taking this serious.
Alex.
PS: this empty POST bug is not the explanation for the session mixup
(yet)? Any ideas on that one?
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
--- jk_ajp_common.c.old Fri Jan 23 16:26:12 2004
+++ jk_ajp_common.c Fri Jan 23 16:28:02 2004
@@ -963,10 +963,16 @@
* or Tomcat crashed. In any case we cannot recover this.
*/
+// as+gj/20040123: we don't want to try the buggy POST recovery as it
+// seems to repost buggy data
+// BUT: doesn't work!
+// op-recoverable = JK_FALSE;
+
jk_log(l, JK_LOG_DEBUG,
ajp_send_request 2:
- request body to send %d - request body to resend %d\n,
- ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN);
+ request body to send %d - request body to resend %d/%d\n,
+ ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN,
+ jk_b_get_len(op-post) - AJP_HEADER_LEN);
/*
* POST recovery job is done here.
@@ -977,10 +983,18 @@
* remote Tomcat
*/
if (jk_b_get_len(op-post) AJP_HEADER_LEN) {
-if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
-jk_log(l, JK_LOG_ERROR, Error resending request body\n);
-return JK_FALSE;
-}
+ // as+gj/20040123: seems to be buggy, doesn't fit the debug output
+ // above, customers complain about seeing wrong user's data when doing
+ // a login-POST
+ // if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
+ // jk_log(l, JK_LOG_ERROR, Error resending request body\n);
+ // return JK_FALSE;
+ // }
+
+ // as+gj/20040123 recovery is disabled above shoudn't happen!
+ jk_log(l, JK_LOG_ERROR, Shouldn't be here resending request body! (as+gj/20040123)\n);
+ op-recoverable = JK_FALSE;
+ return JK_FALSE;
}
else {
/* We never sent any POST data and we check if we have to send at
@@ -1009,6 +1023,8 @@
s-content_read = len;
if (!ajp_connection_tcp_send_message(ae, op-post, l)) {
jk_log(l, JK_LOG_ERROR, Error sending request body\n);
+// as+gj/20040123
+op-recoverable = JK_FALSE;
return JK_FALSE;
}
}
@@ -1279,6 +1295,9 @@
* receiving upload data and we must consider that
* operation is no more recoverable
*/
+// as+gj/20040123:
+// don't want to recover
+op-recoverable = JK_FALSE;
if (! op-recoverable) {
*is_recoverable_error = JK_FALSE;
jk_log(l, JK_LOG_ERROR,
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Hello Henri, Am Fr, den 06.02.2004 schrieb Henri Gomez um 10:15: Strange, also I need a complete mod_jk.log for study. OK, here is the first part of my tests with mod_jk from CVS: Test 1: * run apache/mod_jk with cluster and failover (config included) * tc1 is offline * tc2 is a real tomcat After the apache start the first request (post of login.html, included) is directed to tc1, but it doesn't response. The request is directed to tc2, and EchoFilter.java echos the POST-data to the client. This is OK! Trace: jk.log.onlytc2runs Test 2: * run apache/mod_jk with cluster and failover (config included) * tc1 is a simple netcat (nc -l -p 9010 localhost) * tc2 is a real tomcat After the apache starts the first request (post of login.html, included) is directed to tc1. netcat answers the connection and prints out some garbage on the console. Then you kill netcat (^C). Then mod_jk redirects the request to tc2. == here mod_jk forgets the POST-data and no data is shown to the client! Trace: jk.log.netcatdiedfailovertotc2 Tomorrow I'll trace the problem when tc1 fails after it has already sent header and data to the client. Alex. -- Alexander Schwartz ([EMAIL PROTECTED]) http://www.ahus1.de [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (172)]: Into jk_uri_worker_map_t::uri_worker_map_alloc [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (375)]: Into jk_uri_worker_map_t::uri_worker_map_open [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (396)]: jk_uri_worker_map_t::uri_worker_map_open, rule map size is 1 [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (321)]: Into jk_uri_worker_map_t::uri_worker_map_open, match rule /JOBa1822Web/=lb was added [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (408)]: Into jk_uri_worker_map_t::uri_worker_map_open, there are 1 rules [Mon Feb 09 22:36:01 2004] [jk_uri_worker_map.c (422)]: jk_uri_worker_map_t::uri_worker_map_open, done [Mon Feb 09 22:36:01 2004] [jk_worker.c (88)]: Into wc_open [Mon Feb 09 22:36:01 2004] [jk_worker.c (222)]: Into build_worker_map, creating 3 workers [Mon Feb 09 22:36:01 2004] [jk_worker.c (228)]: build_worker_map, creating worker tc1 [Mon Feb 09 22:36:01 2004] [jk_worker.c (148)]: Into wc_create_worker [Mon Feb 09 22:36:01 2004] [jk_worker.c (162)]: wc_create_worker, about to create instance tc1 of ajp13 [Mon Feb 09 22:36:01 2004] [jk_ajp13_worker.c (108)]: Into ajp13_worker_factory [Mon Feb 09 22:36:01 2004] [jk_worker.c (171)]: wc_create_worker, about to validate and init tc1 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1468)]: Into jk_worker_t::validate [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1489)]: In jk_worker_t::validate for worker tc1 contact is localhost:9010 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1522)]: Into jk_worker_t::init [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1542)]: In jk_worker_t::init, setting socket timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1549)]: In jk_worker_t::init, setting socket keepalive to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1556)]: In jk_worker_t::init, setting cache timeout to 15 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1563)]: In jk_worker_t::init, setting connect timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1570)]: In jk_worker_t::init, setting reply timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1577)]: In jk_worker_t::init, setting prepost timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_worker.c (187)]: wc_create_worker, done [Mon Feb 09 22:36:01 2004] [jk_worker.c (238)]: build_worker_map, removing old tc1 worker [Mon Feb 09 22:36:01 2004] [jk_worker.c (228)]: build_worker_map, creating worker tc2 [Mon Feb 09 22:36:01 2004] [jk_worker.c (148)]: Into wc_create_worker [Mon Feb 09 22:36:01 2004] [jk_worker.c (162)]: wc_create_worker, about to create instance tc2 of ajp13 [Mon Feb 09 22:36:01 2004] [jk_ajp13_worker.c (108)]: Into ajp13_worker_factory [Mon Feb 09 22:36:01 2004] [jk_worker.c (171)]: wc_create_worker, about to validate and init tc2 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1468)]: Into jk_worker_t::validate [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1489)]: In jk_worker_t::validate for worker tc2 contact is localhost:9009 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1522)]: Into jk_worker_t::init [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1542)]: In jk_worker_t::init, setting socket timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1549)]: In jk_worker_t::init, setting socket keepalive to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1556)]: In jk_worker_t::init, setting cache timeout to 15 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1563)]: In jk_worker_t::init, setting connect timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1570)]: In jk_worker_t::init, setting reply timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_ajp_common.c (1577)]: In jk_worker_t::init, setting prepost timeout to 0 [Mon Feb 09 22:36:01 2004] [jk_worker.c (187)]: wc_create_worker,
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Am Fr, den 06.02.2004 schrieb Henri Gomez um 10:13: Could you try with the latest code from CVS, since your patch seems to be for an older version ? I've tested with mod_jk 1.2.4/1.2.5, but I will do it again with the latest Version from CVS with jk.log in DEBUG mode. Will be first thing Monday morning. I'd like to see test case and dumps. What is a dump? What are your requirements for a good test case? I also wonder why the tomcat failed since using the prepost_timeout ping/pong could help you there. It would help, but it has not been released yet (mod_jk 1.2.5 is latest released version, but it will be in mod_jk 1.2.6) -- but it doesn't solve problems when a tomcat fails while processing the request. netcat should handle AJP13, not HTTP, so it should be 8009. Sure, yes, it was listening on 8009 and was receiving AJP13. Sorry. Questions : - Did Tomcat failed while handling the request or was allready broken before ? - Did you clients used chunked transfert ? - You say that sometimes after a POST-failover body data from an old request is sent to the other tomcat when the first connection failed. Do you means that the other tomcat get POST DATA which was never sent by the browser in this request ? Strange since the op-post is reset each time we service a new request. As far as I was able to trace when the session mixup happend tomcat was processing data that has been sent about 15 minutes earlier and delivered the result to another user. I think this was due to the fact that mod_jk mixed up the POST data. I don't know about chunked transfers, but this is unlikely as the data was a normal form with data less than 100 bytes. What I was able to test was empty POST after the first tomcat failed while processing the request and mod_jk retried on another tomcat. Wait for my jk.logs. Alex. -- Alexander Schwartz ([EMAIL PROTECTED]) http://www.ahus1.de - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Alexander Schwartz a écrit : Hello Henri, hello mod_jk-Developers, Am Di, den 27.01.2004 schrieb Henri Gomez um 18:00: we are currently experiencing some problems with mod_jk with loadbalancing: (A) when a post-request fails while receiving data from tomcat, the loadbalancer tries to send the request to the other tomcat, but forgets the post body content (i.e. login and password submitted by the user). The size of the body content is only a few bytes ( 100), so it's not the known problem with bodies longer than 8k. (B) it seems that sometimes after a POST-failover body data from an old request is sent to the other tomcat when the first connection failed. This leads to a session mix-up user mix-up! Could you try with the latest code from CVS, since your patch seems to be for an older version ? From what I see in the code the datas in op-post are sent to the new tomcat so I don't see where can be the problem. I'd like to see test case and dumps. In fact we may have to rewrite the ajp_service(): 1) Init Buffers request, post, reply. 2) Feed request buffer and if needed post buffer. 3) Iterate here : 4) Try to send request and if needed post datas to a tomcat. 5) Swith to another tomcat and retry. A problem in such case is that if there is many POST datas we'll have a HUGE POST buffer and that datas won't be send to tomcat immediatly. So a consensus should be found here : - Make the web-server buffering the request/post, and then send to tomcat's. - Keep the current behaviour, where jk feed the data to tomcat as soon as he get them. I also wonder why the tomcat failed since using the prepost_timeout ping/pong could help you there. Questions : - Did Tomcat failed while handling the request or was allready broken before ? - Did you clients used chunked transfert ? - You say that sometimes after a POST-failover body data from an old request is sent to the other tomcat when the first connection failed. Do you means that the other tomcat get POST DATA which was never sent by the browser in this request ? Strange since the op-post is reset each time we service a new request. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Alexander Schwartz a écrit : I'd really like to use mod_jk open source software in our production environment, but I need to make sure it's strong enough for production. I need some advice on the two issues (as far as I can see there has been no commit on CVS and/or discussion results) * do you think the POST-recovery caused the session mix-up? * does our patch successfully eliminate the problem by disabling POST-recovery? I'd like to see this recovery fixed instead of removing it. but I need something working now :( -- thefore I asked if my code is a working, although i know it is not perfect seen with your eyes. The idea is to use buffers to store request datas (headers/datas). OK, understood. What's the exact problem. You send request (headers / datas) to tomcat1. This one fail when sending reply or didn't reply at all ? I don't know when the session mix up happens, and I was unable to reproduce it. But I was to reproduce the following: (a) mod_jk sends the request and data to tomcat1. Tomcat1 closes the connection. mod_jk tries to recover and sends the request with no data to the second tomcat. Sending no data is definitely wrong. I also argue that this is dangerous as a transaction may be triggered twice by sending the request a second time. Could you send a complete mod_jk.log in DEBUG mode ? Use netcat (nc -l -p 8080) to simulate a tomcat1 on port 8080 receiving the headers. Press ^C to simulate the failure of tomcat1. Use a real tomcat2 for failover. netcat should handle AJP13, not HTTP, so it should be 8009. (b) mod_jk sends the request and data to tomcat1. mod_jk receives header and data from tomcat, but the response is not yet complete. tomcat1 closes the connection. mod_jk tries to recover and sends the request with no data to the second tomcat. The output of the second tomcat is appended to the output of the first tomcat. This is definitely wrong. Strange, also I need a complete mod_jk.log for study. I used the attached filter to set up a test case for this. Use two tomcats, and shut down the first tomcat when it is processing its request. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Alexander Schwartz a écrit :
Hello Henri, hello mod_jk-Developers,
Am Di, den 27.01.2004 schrieb Henri Gomez um 18:00:
I'd really like to use mod_jk open source software in our production
environment, but I need to make sure it's strong enough for production.
I need some advice on the two issues (as far as I can see there has been
no commit on CVS and/or discussion results)
* do you think the POST-recovery caused the session mix-up?
* does our patch successfully eliminate the problem by disabling
POST-recovery?
I'd like to see this recovery fixed instead of removing it.
The idea is to use buffers to store request datas (headers/datas).
What's the exact problem.
You send request (headers / datas) to tomcat1.
This one fail when sending reply or didn't reply at all ?
--- jk_ajp_common.c.old Fri Jan 23 16:26:12 2004
+++ jk_ajp_common.c Fri Jan 23 16:28:02 2004
@@ -963,10 +963,16 @@
* or Tomcat crashed. In any case we cannot recover this.
*/
+// as+gj/20040123: we don't want to try the buggy POST recovery as it
+// seems to repost buggy data
+// BUT: doesn't work!
+// op-recoverable = JK_FALSE;
+
jk_log(l, JK_LOG_DEBUG,
ajp_send_request 2:
- request body to send %d - request body to resend %d\n,
- ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN);
+ request body to send %d - request body to resend %d/%d\n,
+ ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN,
+ jk_b_get_len(op-post) - AJP_HEADER_LEN);
/*
* POST recovery job is done here.
@@ -977,10 +983,18 @@
* remote Tomcat
*/
if (jk_b_get_len(op-post) AJP_HEADER_LEN) {
-if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
-jk_log(l, JK_LOG_ERROR, Error resending request body\n);
-return JK_FALSE;
-}
+ // as+gj/20040123: seems to be buggy, doesn't fit the debug output
+ // above, customers complain about seeing wrong user's data when doing
+ // a login-POST
+ // if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
+ // jk_log(l, JK_LOG_ERROR, Error resending request body\n);
+ // return JK_FALSE;
+ // }
+
+ // as+gj/20040123 recovery is disabled above shoudn't happen!
+ jk_log(l, JK_LOG_ERROR, Shouldn't be here resending request body! (as+gj/20040123)\n);
+ op-recoverable = JK_FALSE;
+ return JK_FALSE;
}
else {
/* We never sent any POST data and we check if we have to send at
@@ -1009,6 +1023,8 @@
s-content_read = len;
if (!ajp_connection_tcp_send_message(ae, op-post, l)) {
jk_log(l, JK_LOG_ERROR, Error sending request body\n);
+// as+gj/20040123
+op-recoverable = JK_FALSE;
return JK_FALSE;
}
}
@@ -1279,6 +1295,9 @@
* receiving upload data and we must consider that
* operation is no more recoverable
*/
+// as+gj/20040123:
+// don't want to recover
+op-recoverable = JK_FALSE;
if (! op-recoverable) {
*is_recoverable_error = JK_FALSE;
jk_log(l, JK_LOG_ERROR,
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
I'd really like to use mod_jk open source software in our production
environment, but I need to make sure it's strong enough for production.
I need some advice on the two issues (as far as I can see there has been
no commit on CVS and/or discussion results)
* do you think the POST-recovery caused the session mix-up?
* does our patch successfully eliminate the problem by disabling
POST-recovery?
I'd like to see this recovery fixed instead of removing it.
but I need something working now :( -- thefore I asked if my code is a
working, although i know it is not perfect seen with your eyes.
The idea is to use buffers to store request datas (headers/datas).
OK, understood.
What's the exact problem.
You send request (headers / datas) to tomcat1.
This one fail when sending reply or didn't reply at all ?
I don't know when the session mix up happens, and I was unable to
reproduce it. But I was to reproduce the following:
(a) mod_jk sends the request and data to tomcat1. Tomcat1 closes the
connection. mod_jk tries to recover and sends the request with no data
to the second tomcat. Sending no data is definitely wrong. I also argue
that this is dangerous as a transaction may be triggered twice by
sending the request a second time.
Use netcat (nc -l -p 8080) to simulate a tomcat1 on port 8080
receiving the headers. Press ^C to simulate the failure of tomcat1. Use
a real tomcat2 for failover.
(b) mod_jk sends the request and data to tomcat1. mod_jk receives
header and data from tomcat, but the response is not yet complete.
tomcat1 closes the connection. mod_jk tries to recover and sends the
request with no data to the second tomcat. The output of the second
tomcat is appended to the output of the first tomcat. This is definitely
wrong.
I used the attached filter to set up a test case for this. Use two
tomcats, and shut down the first tomcat when it is processing its
request.
Alex.
--- jk_ajp_common.c.old Fri Jan 23 16:26:12 2004
+++ jk_ajp_common.c Fri Jan 23 16:28:02 2004
@@ -963,10 +963,16 @@
* or Tomcat crashed. In any case we cannot recover this.
*/
+// as+gj/20040123: we don't want to try the buggy POST recovery as it
+// seems to repost buggy data
+// BUT: doesn't work!
+// op-recoverable = JK_FALSE;
+
jk_log(l, JK_LOG_DEBUG,
ajp_send_request 2:
- request body to send %d - request body to resend %d\n,
- ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN);
+ request body to send %d - request body to resend %d/%d\n,
+ ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN,
+ jk_b_get_len(op-post) - AJP_HEADER_LEN);
/*
* POST recovery job is done here.
@@ -977,10 +983,18 @@
* remote Tomcat
*/
if (jk_b_get_len(op-post) AJP_HEADER_LEN) {
-if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
-jk_log(l, JK_LOG_ERROR, Error resending request body\n);
-return JK_FALSE;
-}
+ // as+gj/20040123: seems to be buggy, doesn't fit the debug output
+ // above, customers complain about seeing wrong user's data when doing
+ // a login-POST
+ // if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
+ // jk_log(l, JK_LOG_ERROR, Error resending request body\n);
+ // return JK_FALSE;
+ // }
+
+ // as+gj/20040123 recovery is disabled above shoudn't happen!
+ jk_log(l, JK_LOG_ERROR, Shouldn't be here resending request body!
(as+gj/20040123)\n);
+ op-recoverable = JK_FALSE;
+ return JK_FALSE;
}
else {
/* We never sent any POST data and we check if we have to send at
@@ -1009,6 +1023,8 @@
s-content_read = len;
if (!ajp_connection_tcp_send_message(ae, op-post, l)) {
jk_log(l, JK_LOG_ERROR, Error sending request body\n);
+// as+gj/20040123
+op-recoverable = JK_FALSE;
return JK_FALSE;
}
}
@@ -1279,6 +1295,9 @@
* receiving upload data and we must consider that
* operation is no more recoverable
*/
+// as+gj/20040123:
+// don't want to recover
+op-recoverable = JK_FALSE;
if (! op-recoverable) {
*is_recoverable_error = JK_FALSE;
jk_log(l, JK_LOG_ERROR,
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
package com.direkt1822.joba.filter;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.PrintWriter;
import javax.servlet.Filter;
import
Re: failover-problem and session mixup: jakarta-tomcat-connectors/mod_jk: jk_ajp_common.c
Hello Henri, hello mod_jk-Developers,
Am Di, den 27.01.2004 schrieb Henri Gomez um 18:00:
we are currently experiencing some problems with mod_jk with
loadbalancing:
(A) when a post-request fails while receiving data from tomcat, the
loadbalancer tries to send the request to the other tomcat, but
forgets the post body content (i.e. login and password submitted by
the user). The size of the body content is only a few bytes ( 100), so
it's not the known problem with bodies longer than 8k.
(B) it seems that sometimes after a POST-failover body data from an
old request is sent to the other tomcat when the first connection
failed. This leads to a session mix-up user mix-up!
[...]
It used to works but it was before the content-header add-on code.
This piece of is a nightmare, and I see no easier solution than copying
the HEAD+POST in temp buffer but I'd like to have Bill, Mladen and JFC
advices...
I'd really like to use mod_jk open source software in our production
environment, but I need to make sure it's strong enough for production.
I need some advice on the two issues (as far as I can see there has been
no commit on CVS and/or discussion results)
* do you think the POST-recovery caused the session mix-up?
* does our patch successfully eliminate the problem by disabling
POST-recovery?
The patch is attached.
Thanks a lot,
Alexander.
--
Alexander Schwartz ([EMAIL PROTECTED])
http://www.ahus1.de
--- jk_ajp_common.c.old Fri Jan 23 16:26:12 2004
+++ jk_ajp_common.c Fri Jan 23 16:28:02 2004
@@ -963,10 +963,16 @@
* or Tomcat crashed. In any case we cannot recover this.
*/
+// as+gj/20040123: we don't want to try the buggy POST recovery as it
+// seems to repost buggy data
+// BUT: doesn't work!
+// op-recoverable = JK_FALSE;
+
jk_log(l, JK_LOG_DEBUG,
ajp_send_request 2:
- request body to send %d - request body to resend %d\n,
- ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN);
+ request body to send %d - request body to resend %d/%d\n,
+ ae-left_bytes_to_send, jk_b_get_len(op-reply) - AJP_HEADER_LEN,
+ jk_b_get_len(op-post) - AJP_HEADER_LEN);
/*
* POST recovery job is done here.
@@ -977,10 +983,18 @@
* remote Tomcat
*/
if (jk_b_get_len(op-post) AJP_HEADER_LEN) {
-if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
-jk_log(l, JK_LOG_ERROR, Error resending request body\n);
-return JK_FALSE;
-}
+ // as+gj/20040123: seems to be buggy, doesn't fit the debug output
+ // above, customers complain about seeing wrong user's data when doing
+ // a login-POST
+ // if(!ajp_connection_tcp_send_message(ae, op-post, l)) {
+ // jk_log(l, JK_LOG_ERROR, Error resending request body\n);
+ // return JK_FALSE;
+ // }
+
+ // as+gj/20040123 recovery is disabled above shoudn't happen!
+ jk_log(l, JK_LOG_ERROR, Shouldn't be here resending request body! (as+gj/20040123)\n);
+ op-recoverable = JK_FALSE;
+ return JK_FALSE;
}
else {
/* We never sent any POST data and we check if we have to send at
@@ -1009,6 +1023,8 @@
s-content_read = len;
if (!ajp_connection_tcp_send_message(ae, op-post, l)) {
jk_log(l, JK_LOG_ERROR, Error sending request body\n);
+// as+gj/20040123
+op-recoverable = JK_FALSE;
return JK_FALSE;
}
}
@@ -1279,6 +1295,9 @@
* receiving upload data and we must consider that
* operation is no more recoverable
*/
+// as+gj/20040123:
+// don't want to recover
+op-recoverable = JK_FALSE;
if (! op-recoverable) {
*is_recoverable_error = JK_FALSE;
jk_log(l, JK_LOG_ERROR,
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
