RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Henri, Dan,
There are some discussions below about the usage
of msg/rmsg. What did you guys finally decide
on that? Just reuse msg? I'm seeing the problem
Dan mentioned on message larger than the ajp13
buffer (8K).
thanks,
shinta
-Original Message-
From: GOMEZ Henri [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, March 13, 2001 4:49 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
1) First off, I strongly believe that this work (which I think is an
excellent feature, BTW), belongs in the 3.3 branch, and
*not* in 3.2.
+1, there was a message about including latest mod_jk
features/corrections
back to Tomcat 3.2.x but TC 3.2 Release Manager and others clairly
stated that TC 3.2.x must be just correction of bugs without any
new features. I asked about back-porting mod_jk/ajp13 since now
Apache 2.0 or upload works fine with mod_jk/ajp from TC 3.3 branches
but are not in TC 3.2. The decision was not to provide corrections
to TC 3.2.
Although it feels like a bug that we need to restart Apache
every time we restart TC, it's documented behavior, and I
consider fixing it more of a feature enhancement.
The scope of work you're doing here is considerable,
and the code you're modifying is complex (and involves lots of
different possible situations). I would not feel
comfortable committing
this to 3.2 unless it had seen a *lot* of testing in the 3.3
branch. (Am
I incorrect in remembering that there has been discussion of
putting this
into 3.2?)
Right many works, some testing but back-port could introduce
bugs in TC3.2.
The decision was to not port mod_jk back to tc 3.2. But
people could still
build mod_jk from Tomcat 3.3 and use against there Tomcat
3.2.x. That's why
I milite for a jakarta sub-project (web-connector). These
kind of connector
work must outside Tomcat trees. (Votes ?)
2) Specific problems (line numbers by patch against the 3.3
code base)
- msg/rmsg: Here is how I think you want the code to work:
msg always holds the basic request information, so, in case
of an error, we
can
resend the message. rmsg is used as the buffer for response
information.
Dan you're our resident hacker/expert on mod_jk native part
This is a good idea, but there are a few problems -- first, and most
importantly, if there is POST data, the msg buffer
immediately gets reused to send that information to TC (ll.
596-600).
In that case, on line 691, if you retry after a connection
reset event,
you'd be using a buffer filled with POST data instead of a
buffer with
header information.
I'm not too confortable with msg/rmsg that's why I asked for your code
revue (and it's excellent ;-)
Possibly you could fix this by calling
ajp13_marshal_into_msgb again when
you retry -- I don't know how the state of the
web_server_service object
(s),
changes over time, so I don't know for sure if this would
work or not. But
that's where I'd look next.
Or you could pass rmsg into the sendrequest, and use that to
send the POST data over. I dunno.
Your use of msg/rmsg in getreply seems faulty to me. I don't
think you need msg there at all, and I'm sure that you've
got a problem
with
mixing the two of them. Specifically, on l. 636, you call
rc = ajp13_process_callback(rmsg, p, s, l);
But, then a few lines later, you do:
else if(JK_AJP13_HAS_RESPONSE == rc) {
rc = connection_tcp_send_message(p, msg, l);
Right, we must drop rmsg use and do all stuff with msg.
If you look at ajp13_process_callback, at the GET_BODY_CHUNK
case, you'll see that it's reading request data into the
buffer passed in
as a param
(which is what getreply is calling 'rmsg'). But then, when you call
connection_tcp_send_message, you'll be sending over whatever
was in msg, rather than what the web server has read from
the browser.
You can fix this by changing connection_tcp_send_message to
use 'rmsg' (and
then you don't need to pass msg into getreply at all).
This problem will happen in case of a file upload, or
whenever there is
enough POST data to exceed a single ajp13 buffer (8K).
Furthermore, in that
case, I'm not sure if there's going to be any way to restart
the connection
intelligently. If the browser has already sent over 500 K
of a 1M file
upload when TC dies, I think we need to just kill that
request, rather than
trying to restart it. I don't think the request is
recoverable. I'm not sure
how to detect this in your code, but I think it needs to be
thought about.
Upload are a real problem since we just couldn't replay the upload
in case of Tomcat failure. Solution could be to ask browser to
resend but HOW (HTTP expert needed here)
- The for loop on l. 689 makes me nervous. Could you rewrite that:
int i;
for(i = 0 ; i JK_RETRIES ; i++)
That way
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I proposed to [EMAIL PROTECTED] list some days now that we start a subproject called web-connector where we could put mod_jk/mod_jserv/mod_webapp with both native (Apache, NES, IIS) and Java (Modules/Interceptors for TC 3.2.x, 3.3 and 4.0) Still no response but I really think we must externalize that connector stuff from Tomcat's core code ;-) 1) First off, I strongly believe that this work (which I think is an excellent feature, BTW), belongs in the 3.3 branch, and *not* in 3.2. Although it feels like a bug that we need to restart Apache every time we restart TC, it's documented behavior, and I consider fixing it more of a feature enhancement. The scope of work you're doing here is considerable, and the code you're modifying is complex (and involves lots of different possible situations). I would not feel comfortable committing this to 3.2 unless it had seen a *lot* of testing in the 3.3 branch. (Am I incorrect in remembering that there has been discussion of putting this into 3.2?) Dan, sorry for not clarifying. I'm back porting the changes to my own company's CVS, not the Apache Tomcat's CVS. We have decided that we need this changes on our 3.2.1 Tomcat release but can't afford to release Tomcat 3.3 beta right now. So you can definitely check in the changes to 3.3 branch, and not 3.2.1. I think we've discussed that and that was agreed. thanks, shinta
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
In the mean time, I have taken Henri's changes and back port it to 3.2.1 (because I need it on 3.2.1). Everything seems to work well. I've tested it in the normal scenarios (one Apache, one Tomcat) and in the load-balanced scenarios. Thanks for using the patch and tested it under LoadBalanced Tomcats. In the load-balanced scenarios, when I restart TC worker 1, the code properly close the dead sockets and re-establish new ones to the same worker (TC worker 1). The good connections to TC worker 2 are untouched. They stay connected. Normal procedure I did notice something wierd. But this is un-related to the code edits. This happens with or without Henri's changes. When I restart TC worker 1, but shut down TC worker 2, requests that supposed to go to TC worker 2 (because they belong to the same session, thus the load balancer try to foward it to the same TC worker 2) took sometime to get forwarded to TC worker 1. This maybe another one of those "improvements" that can be done to the load balancer worker. No problem here, when you shutdown a Tomcat in a load balancing architecture, you got request goes to the second even if there is a JVMROUTE set . Anyway, I'm pretty happy with Henri's changes. (Thanks Henri!). Henri, are you going to check in the changes? I'd like to see Dan check it since I created a second memory pool rmsg but I'm not too confident on it : +rmsg = jk_b_new((p-pool)); +jk_b_set_buffer_size( rmsg, DEF_BUFFER_SZ); + jk_b_reset(rmsg); See you - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
Re: FW: problem w/ ajp13 - if Tomcat is shutdown
Henri, I think you're right about not trying to clean out the cache of endpoints. Too much work and not really necessary. I'd be happy to check your code, but I don't think I've seen it -- did you submit it to the list (I may have missed a few messages to the list)? -Dan GOMEZ Henri wrote: In the mean time, I have taken Henri's changes and back port it to 3.2.1 (because I need it on 3.2.1). Everything seems to work well. I've tested it in the normal scenarios (one Apache, one Tomcat) and in the load-balanced scenarios. Thanks for using the patch and tested it under LoadBalanced Tomcats. In the load-balanced scenarios, when I restart TC worker 1, the code properly close the dead sockets and re-establish new ones to the same worker (TC worker 1). The good connections to TC worker 2 are untouched. They stay connected. Normal procedure I did notice something wierd. But this is un-related to the code edits. This happens with or without Henri's changes. When I restart TC worker 1, but shut down TC worker 2, requests that supposed to go to TC worker 2 (because they belong to the same session, thus the load balancer try to foward it to the same TC worker 2) took sometime to get forwarded to TC worker 1. This maybe another one of those "improvements" that can be done to the load balancer worker. No problem here, when you shutdown a Tomcat in a load balancing architecture, you got request goes to the second even if there is a JVMROUTE set . Anyway, I'm pretty happy with Henri's changes. (Thanks Henri!). Henri, are you going to check in the changes? I'd like to see Dan check it since I created a second memory pool rmsg but I'm not too confident on it : +rmsg = jk_b_new((p-pool)); +jk_b_set_buffer_size( rmsg, DEF_BUFFER_SZ); + jk_b_reset(rmsg); See you - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED] -- Dan Milstein // [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
Re: FW: problem w/ ajp13 - if Tomcat is shutdown
Henri,
I looked in the archives and found the patch which I hadn't seen (my ISP's
mail server needs a serious talking to!).
Okay, I think I see some fairly serious issues. I'll lay out here what I've
seen so far:
1) First off, I strongly believe that this work (which I think is an
excellent feature, BTW), belongs in the 3.3 branch, and *not* in 3.2.
Although it feels like a bug that we need to restart Apache every time we
restart TC, it's documented behavior, and I consider fixing it more of a
feature enhancement. The scope of work you're doing here is considerable,
and the code you're modifying is complex (and involves lots of different
possible situations). I would not feel comfortable committing this to 3.2
unless it had seen a *lot* of testing in the 3.3 branch. (Am I incorrect in
remembering that there has been discussion of putting this into 3.2?)
2) Specific problems (line numbers by patch against the 3.3 code base)
- msg/rmsg: Here is how I think you want the code to work: msg always holds
the basic request information, so, in case of an error, we can resend the
message. rmsg is used as the buffer for response information.
This is a good idea, but there are a few problems -- first, and most
importantly, if there is POST data, the "msg" buffer immediately gets reused
to send that information to TC (ll. 596-600). In that case, on line 691, if
you retry after a connection reset event, you'd be using a buffer filled
with POST data instead of a buffer with header information. Possibly you
could fix this by calling ajp13_marshal_into_msgb again when you retry -- I
don't know how the state of the web_server_service object (s), changes over
time, so I don't know for sure if this would work or not. But that's where
I'd look next. Or you could pass rmsg into the sendrequest, and use that to
send the POST data over. I dunno.
Your use of msg/rmsg in getreply seems faulty to me. I don't think you need
msg there at all, and I'm sure that you've got a problem with mixing the two
of them. Specifically, on l. 636, you call
rc = ajp13_process_callback(rmsg, p, s, l);
But, then a few lines later, you do:
else if(JK_AJP13_HAS_RESPONSE == rc) {
rc = connection_tcp_send_message(p, msg, l);
If you look at ajp13_process_callback, at the GET_BODY_CHUNK case, you'll
see that it's reading request data into the buffer passed in as a param
(which is what getreply is calling 'rmsg'). But then, when you call
connection_tcp_send_message, you'll be sending over whatever was in msg,
rather than what the web server has read from the browser. You can fix this
by changing connection_tcp_send_message to use 'rmsg' (and then you don't
need to pass msg into getreply at all).
This problem will happen in case of a file upload, or whenever there is
enough POST data to exceed a single ajp13 buffer (8K). Furthermore, in that
case, I'm not sure if there's going to be any way to restart the connection
intelligently. If the browser has already sent over 500 K of a 1M file
upload when TC dies, I think we need to just kill that request, rather than
trying to restart it. I don't think the request is recoverable. I'm not sure
how to detect this in your code, but I think it needs to be thought about.
- The for loop on l. 689 makes me nervous. Could you rewrite that:
int i;
for(i = 0 ; i JK_RETRIES ; i++)
That way you won't get an infinite loop if JK_RETRIES 0. And I think it's
more idiomatic. But, hey, that's up to you ;-).
- Do you understand the JK_METHOD macro? I don't fully, but take a look in
jk_global.h. In Win32 and Netware environments, this macro is set to
"__stdcall" -- is this only needed for things which are called from outside
this file (like service())? If neither of us understand it, I'd like to
have people on Win32 and Netware systems test this (maybe they could even
explain why that call is needed -- and then we could document it!) Maybe
you should be using it with sendrequest and getreply?
- A few logging inconsistencies -- you call jk_log(l, JK_LOG_DEBUG,...),
when I think you want JK_LOG_ERROR. Check l. 642, 702.
- Error/log message inconsistencies:
- l. 632 You mean "error reading response", I think
- l. 642 You have "1" where you want ":", I think.
- You seem to be using 4-character tabs -- the rest of the C-code (AFAIK)
uses 8-character tabs. I think that's our standard (it is on the Java side,
for sure). In any event, this is making your new code render bizarrely in
my editor.
- Style notes (I'm style obsessive, I admit). I would make it
ajp13_send_request (or at least send_request) and ajp13_get_reply (or
get_reply).
-Dan
GOMEZ Henri wrote:
In the mean time, I have taken Henri's changes and back
port it to 3.2.1 (because I need it on 3.2.1). Everything
seems to work well. I've tested it in the normal scenarios
(one Apache, one Tomcat) and in the load-balanced scenarios.
Thanks for using the patch and tested it under LoadBalanced
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown 1) First off, I strongly believe that this work (which I think is an excellent feature, BTW), belongs in the 3.3 branch, and *not* in 3.2. Although it feels like a bug that we need to restart Apache every time we restart TC, it's documented behavior, and I consider fixing it more of a feature enhancement. The scope of work you're doing here is considerable, and the code you're modifying is complex (and involves lots of different possible situations). I would not feel comfortable committing this to 3.2 unless it had seen a *lot* of testing in the 3.3 branch. (Am I incorrect in remembering that there has been discussion of putting this into 3.2?) Dan, sorry for not clarifying. I'm back porting the changes to my own company's CVS, not the Apache Tomcat's CVS. We have decided that we need this changes on our 3.2.1 Tomcat release but can't afford to release Tomcat 3.3 beta right now. So you can definitely check in the changes to 3.3 branch, and not 3.2.1. I think we've discussed that and that was agreed. thanks, shinta
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Here how we could do : Okay, I basically agree with you. I'll take out the check for errno and just have recv() == -1 be considered a recoverable error (i.e: retry it). However, I disagree with making the retry in a loop for RETRIES times. This is because if one retry fails, this means this error condition may not be recoverable without any human interventions. We must retry against eventual other workers in a LoadBalancing system. I must verify that my patch allow that. What is the point of retrying more than once? No problem, I'll retry 3 times. It's a special case, exception, so we could spend some time to re-establish the connection to a working unit. My goal is not to wait for TC to come back up or to wait for TC to be in a good state. My goal is, if TC is in a good state already, why tell the caller that it's an error. +1 Opinions? I agree, with the new version you could see that we try X time to send the request. It wasn't the case previously which make me think that load-balancing was only working on active server, but failed completly when one of then was stopped. We jk_ajp13_worker.c.diff - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
La prise de conscience de votre propre ignorance est un grand pas vers la connaissance. -- Benjamin Disraeli -Original Message- From: Dan Milstein [mailto:[EMAIL PROTECTED]] Sent: Friday, March 09, 2001 6:34 AM To: [EMAIL PROTECTED] Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown Henri, You say that checking errno isn't safe in a multithreaded env (which would certainly makes sense to me, since it looks like a global var). However, after searching online, and reading up in "Programming Threads", by Kleiman, Shah and Smaalders, I find on p. 47: "Each thread has its own independent version of the errno variable. This allows different threads to make system calls that may change the value of errno without interfering with each other." They are describing Posix threads. "errno" is actually a macro, apparently, which accesses the correct, thread-specific errno variable. Right, I checked in Linux errno.h for pthread Now, I am the first to admit that I don't understand all the weird intersections between threads and sockets in C, but this looks to me like checking errno against ECONNRESET may be fine. More generally when you got a read error on TCP/IP stream you could consider that the link to your server (tomcat) is broken : - no more route to tomcat (broken lan or routers) - server not working (tomcat was stopped or server restarted) Are there platforms where that's not true? I've no idea but we migth have problems in differents interpretation of platform. The nice thing about getting that ECONNRESET error, is it lets us go ahead and close out that connection, and try another one. Done. We could even close out a whole cache of connections, which would most likely be the right thing to do. Good idea, I'll find how to do that. If we loop/retry, than how do we know to close the connection? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
Re: FW: problem w/ ajp13 - if Tomcat is shutdown
In terms of invalidating the cache: The jk_ajp13_worker objects hold onto a cache of endpoints -- ep_cache. It would be relatively simple to loop through this cache and close all the connections in case of ECONNRESET (you do have to call a macro to enter a critical section -- take a look at reuse_connection()). However, this cache only holds onto endpoints which are *not* being used. When an endpoint is checked out of the cache (by get_endpoint), or if the open socket descriptor is transfered to another endpoint (in reuse_connection), that connection is replaced by NULL in the cache. So if we shut down all the connections in the cache, we won't shut down the other connections which are handling requests at that moment. My only fear then is that, when those connections get their own ECONNRESET errors, they, too, will try to shutdown all the connections in the cache. If TC hasn't come back up yet, this won't be a problem, because there won't be any connections in the cache. But it does make me a bit nervous. Hope that's helpful... -Dan GOMEZ Henri wrote: La prise de conscience de votre propre ignorance est un grand pas vers la connaissance. -- Benjamin Disraeli -Original Message- From: Dan Milstein [mailto:[EMAIL PROTECTED]] Sent: Friday, March 09, 2001 6:34 AM To: [EMAIL PROTECTED] Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown Henri, You say that checking errno isn't safe in a multithreaded env (which would certainly makes sense to me, since it looks like a global var). However, after searching online, and reading up in "Programming Threads", by Kleiman, Shah and Smaalders, I find on p. 47: "Each thread has its own independent version of the errno variable. This allows different threads to make system calls that may change the value of errno without interfering with each other." They are describing Posix threads. "errno" is actually a macro, apparently, which accesses the correct, thread-specific errno variable. Right, I checked in Linux errno.h for pthread Now, I am the first to admit that I don't understand all the weird intersections between threads and sockets in C, but this looks to me like checking errno against ECONNRESET may be fine. More generally when you got a read error on TCP/IP stream you could consider that the link to your server (tomcat) is broken : - no more route to tomcat (broken lan or routers) - server not working (tomcat was stopped or server restarted) Are there platforms where that's not true? I've no idea but we migth have problems in differents interpretation of platform. The nice thing about getting that ECONNRESET error, is it lets us go ahead and close out that connection, and try another one. Done. We could even close out a whole cache of connections, which would most likely be the right thing to do. Good idea, I'll find how to do that. If we loop/retry, than how do we know to close the connection? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED] -- Dan Milstein // [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown Hi, Dan, I may not understand all of the issues here. But I really don't think we should close all connections when we detect one ECONNRESET. In my opinion, it is not necessary to close all connections, since the next ECONNRESET will close the proper dead socket, anyway. It's not needed to add all of those complexity. In the mean time, I have taken Henri's changes and back port it to 3.2.1 (because I need it on 3.2.1). Everything seems to work well. I've tested it in the normal scenarios (one Apache, one Tomcat) and in the load-balanced scenarios. In the load-balanced scenarios, when I restart TC worker 1, the code properly close the dead sockets and re-establish new ones to the same worker (TC worker 1). The good connections to TC worker 2 are untouched. They stay connected. I did notice something wierd. But this is un-related to the code edits. This happens with or without Henri's changes. When I restart TC worker 1, but shut down TC worker 2, requests that supposed to go to TC worker 2 (because they belong to the same session, thus the load balancer try to foward it to the same TC worker 2) took sometime to get forwarded to TC worker 1. This maybe another one of those improvements that can be done to the load balancer worker. Anyway, I'm pretty happy with Henri's changes. (Thanks Henri!). Henri, are you going to check in the changes? Let me know if I can do something else to help for this case. shinta -Original Message- From: Dan Milstein [mailto:[EMAIL PROTECTED]] Sent: Friday, March 09, 2001 3:08 PM To: [EMAIL PROTECTED] Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown In terms of invalidating the cache: The jk_ajp13_worker objects hold onto a cache of endpoints -- ep_cache. It would be relatively simple to loop through this cache and close all the connections in case of ECONNRESET (you do have to call a macro to enter a critical section -- take a look at reuse_connection()). However, this cache only holds onto endpoints which are *not* being used. When an endpoint is checked out of the cache (by get_endpoint), or if the open socket descriptor is transfered to another endpoint (in reuse_connection), that connection is replaced by NULL in the cache. So if we shut down all the connections in the cache, we won't shut down the other connections which are handling requests at that moment. My only fear then is that, when those connections get their own ECONNRESET errors, they, too, will try to shutdown all the connections in the cache. If TC hasn't come back up yet, this won't be a problem, because there won't be any connections in the cache. But it does make me a bit nervous. Hope that's helpful... -Dan GOMEZ Henri wrote: La prise de conscience de votre propre ignorance est un grand pas vers la connaissance. -- Benjamin Disraeli -Original Message- From: Dan Milstein [mailto:[EMAIL PROTECTED]] Sent: Friday, March 09, 2001 6:34 AM To: [EMAIL PROTECTED] Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown Henri, You say that checking errno isn't safe in a multithreaded env (which would certainly makes sense to me, since it looks like a global var). However, after searching online, and reading up in Programming Threads, by Kleiman, Shah and Smaalders, I find on p. 47: Each thread has its own independent version of the errno variable. This allows different threads to make system calls that may change the value of errno without interfering with each other. They are describing Posix threads. errno is actually a macro, apparently, which accesses the correct, thread-specific errno variable. Right, I checked in Linux errno.h for pthread Now, I am the first to admit that I don't understand all the weird intersections between threads and sockets in C, but this looks to me like checking errno against ECONNRESET may be fine. More generally when you got a read error on TCP/IP stream you could consider that the link to your server (tomcat) is broken : - no more route to tomcat (broken lan or routers) - server not working (tomcat was stopped or server restarted) Are there platforms where that's not true? I've no idea but we migth have problems in differents interpretation of platform. The nice thing about getting that ECONNRESET error, is it lets us go ahead and close out that connection, and try another one. Done. We could even close out a whole cache of connections, which would most likely be the right thing to do. Good idea, I'll find how to do that. If we loop/retry, than how do we know to close the connection? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED] -- Dan Milstein
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I revue your patch and I notice : We could use select to determine if something happen to the connection, typically readfds will be set if something is to read. With ajp13 protocol where everything is consumed after reading the previous tomcat reply, select will set read when the connection is closed (in TCP a closed tcp session is really a message send from closer). But : - Win32 platform are not handled with the select; - there is a check of errno which couldn't be safe in multi-threaded env like Apache 2.0 - if you remove your ethernet cable you may never see anything in select readdfs before TCP/IP stack timeout. - Why loose cpu cycle to check if the communication socket is available before each request to be sent. I'll be to use an exception mecanism, so - send request - read reply - if reply read fail, retry to send request, if it fail = ERR 500 I'll send some modification to worker later
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I revue your patch and I notice : We could use select to determine if something happen to the connection, typically readfds will be set if something is to read. With ajp13 protocol where everything is consumed after reading the previous tomcat reply, select will set read when the connection is closed (in TCP a closed tcp session is really a message send from closer). But : - Win32 platform are not handled with the select; This is because I haven't been able to reproduce this on Windows 2000. I admit I didn't try very hard last time. So I will try it again. - there is a check of errno which couldn't be safe in multi-threaded env like Apache 2.0 Any suggestions on how to do it better? The reason I check for ECONNRESET is I want to do this only if the error was caused by previously shut TC. Any other recv() error should return Internal Server Error, because they may not be recoverable. - if you remove your ethernet cable you may never see anything in select readdfs before TCP/IP stack timeout. well, I wasn't trying to handle that condition, though. The select() should still be harmless. - Why loose cpu cycle to check if the communication socket is available before each request to be sent. I'll be to use an exception mecanism, so - send request - read reply - if reply read fail, retry to send request, if it fail = ERR 500 Okay so this means, you would prefer my proposed solution #1? That was my inclination too. I'll send some modification to worker later
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Okay so this means, you would prefer my proposed solution #1?
That was my inclination too.
Proposed solution #1 without the errno check.
My idea :
get the service code in a loop
for (i = 0; i RETRIES; i++) {
if (send_request() 0)
continue;
if (read_reply() == 0)
break;
}
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Okay, I basically agree with you. I'll take out
the check for errno and just have recv() == -1
be considered a recoverable error (i.e: retry it).
However, I disagree with making the retry in a
loop for RETRIES times. This is because if one
retry fails, this means this error condition may
not be recoverable without any human interventions.
What is the point of retrying more than once?
My goal is not to wait for TC to come back up
or to wait for TC to be in a good state. My goal
is, if TC is in a good state already, why tell
the caller that it's an error.
Opinions?
shinta
-Original Message-
From: GOMEZ Henri [mailto:[EMAIL PROTECTED]]
Sent: Thursday, March 08, 2001 5:12 PM
To: [EMAIL PROTECTED]
Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Okay so this means, you would prefer my proposed solution #1?
That was my inclination too.
Proposed solution #1 without the errno check.
My idea :
get the service code in a loop
for (i = 0; i RETRIES; i++) {
if (send_request() 0)
continue;
if (read_reply() == 0)
break;
}
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
Re: FW: problem w/ ajp13 - if Tomcat is shutdown
Henri,
You say that checking errno isn't safe in a multithreaded env (which would
certainly makes sense to me, since it looks like a global var).
However, after searching online, and reading up in "Programming Threads", by
Kleiman, Shah and Smaalders, I find on p. 47:
"Each thread has its own independent version of the errno variable. This
allows different threads to make system calls that may change the value of
errno without interfering with each other."
They are describing Posix threads. "errno" is actually a macro, apparently,
which accesses the correct, thread-specific errno variable.
Now, I am the first to admit that I don't understand all the weird
intersections between threads and sockets in C, but this looks to me like
checking errno against ECONNRESET may be fine.
Are there platforms where that's not true?
The nice thing about getting that ECONNRESET error, is it lets us go ahead
and close out that connection, and try another one. We could even close out
a whole cache of connections, which would most likely be the right thing to
do. If we loop/retry, than how do we know to close the connection?
-Dan
GOMEZ Henri wrote:
Okay so this means, you would prefer my proposed solution #1?
That was my inclination too.
Proposed solution #1 without the errno check.
My idea :
get the service code in a loop
for (i = 0; i RETRIES; i++) {
if (send_request() 0)
continue;
if (read_reply() == 0)
break;
}
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
--
Dan Milstein // [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
BTW we should redirect this to the tomcat-users list as I don't think its a development issue. I don't see anything wrong, and the only drastic difference from my previous setup was that I had used the default 8007 and 8009. Now This is purely a guess, but I seem to remember reading that ports 1 were special somehow and required root access for some functions. What I would suggest is put those to the default, see if they work, then (I'm assuming you had a reason for ding this) find an acceptable port between 8000-. If that doesn't work then we have a couple more "generic" possiblities (such as running out of file handles which is very common with tomcat on solaris). I wish I still had access to a solaris box to try this.. The only other difference was that I was using Jdk 1.3 (Sun). Which I'd recommend for server side stuff unless you have a compelling reason not to. Its more stable, has less glitches and is faster. Jdk 1.2.x was suns "wowee" and jdk 1.3.x is a performance and stability release (IMHO).. If you haven't tweaked your file handles at any point then there is a good chance that is in the way, unfortunately I do not remember what the strings were to fix this, but I'm sure their documented somewhere (and they're not intuitive).. Hope this helps some. -Andy -Original Message- From: Shinta Tjio To: 'oliver2, andy'; '[EMAIL PROTECTED] ' Cc: ''Dan Milstein' ' Sent: 3/6/01 7:51 PM Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown Here's exactly what I did to reproduce the problem. Again, I'm running this on a Solaris 2.8 SunOS machine, using JDK1.2.2, Apache 1.3.14, Tomcat 3.2.1. 1. Start Apache 2. Start Tomcat 3. Start hitting Apache with multiple requests, such as /example/servlet/HelloWorldExample. Make sure there are some connections opened from Apache to Tomcat. To make sure, you can do netstat -a | grep Tomcat port where Tomcat port is the ajp13 port. It should show some socket in ESTABLISHED state. 4. While the connections are in the ESTABLISHED state (this should be the state because ajp13 reuse connections), shutdown Tomcat. Now you will notice the same netstat will show some sockets in FIN_WAIT2 and CLOSE_WAIT state. 5. Now restart Tomcat. 6. Repeat step 3. You will get Internal Server Error, up to the number of children Apache has. After that number, the page will be served. Attached is the server.xml, workers.properties, httpd.conf. My test servlet is called /mytest/servlet/testServlet but I think you can try it with any kind of servlet. I have been trying to code up the retry I mentioned below. I think I got it working. I just need to clean up the code some more. shinta -Original Message- From: oliver2, andy [ mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ] Sent: Tuesday, March 06, 2001 6:33 PM To: 'Shinta Tjio '; '[EMAIL PROTECTED] ' Cc: ''Dan Milstein' ' Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I was running recently with that exact same configuration but did not experience that problem. Can you show some config files. -Andy -Original Message- From: Shinta Tjio To: [EMAIL PROTECTED] Cc: 'Dan Milstein' Sent: 3/6/01 7:01 PM Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I am using Tomcat 3.2.1, Apache 1.3.14, running on Solaris 2.8, Sun machines. After various attempts of debugging this, I have more information. 1. Even though I'm setting the worker's property cache_size to default (1), I'm finding there are up to 6 connections opened from Apache to Tomcat. I deduce this by looking at the mod_jk.conf and by doing netstat. I finally find out, this is so because my Apache is set to spawn minimum of 6 children and each of those children are making separate connections to Tomcat. This is very bad because, I ended up having to reload 6 times before Tomcat starts serving me the page again. Each time it uses a different Apache children that has defunct socket. So the more Apache children I have, the longer it takes me to recover from this problem. 2. It seems when Tomcat dies restarts, the send() called by ajp13's jk_tcp_socket_sendfull() does not get an error. But the recv() does get an error, with errno ECONNRESET. After which, the socket is properly closed. 3. When I shutdown Tomcat, those sockets that were opened between Apache/Tomcat showed up in state CLOSE_WAIT, and FIN_WAIT2. I think this is normally solved by calling the shutdown() API after closing the socket. However, this would have to be done from the Tomcat side in Ajp13ConnectionHandler.java. I can't find the corresponding method of Socket in Java. So.. based on all of these, the only fix I can think of putting is to make mod_jk retry the send() if recv() comes back with an error ECONNRESET. The retry should happen after the ol
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown BTW we should redirect this to the tomcat-users list as I don't think its a development issue. Sorry about that. I don't see anything wrong, and the only drastic difference from my previous setup was that I had used the default 8007 and 8009. Now This is purely a guess, but I seem to remember reading that ports 1 were special somehow and required root access for some functions. What I would suggest is put those to the default, see if they work, then (I'm assuming you had a reason for ding this) find an acceptable port between 8000-. I think you get that backwards. Programs listening on port1024 will require root access. Anything larger than that, it doesn't require special priviledge. If that doesn't work then we have a couple more generic possiblities (such as running out of file handles which is very common with tomcat on solaris). I wish I still had access to a solaris box to try this.. The only other difference was that I was using Jdk 1.3 (Sun). Which I'd recommend for server side stuff unless you have a compelling reason not to. Its more stable, has less glitches and is faster. Jdk 1.2.x was suns wowee and jdk 1.3.x is a performance and stability release (IMHO).. If you haven't tweaked your file handles at any point then there is a good chance that is in the way, unfortunately I do not remember what the strings were to fix this, but I'm sure their documented somewhere (and they're not intuitive).. Thanks for the tips. We can't switch JDK for this release we're having. But next release, we are hoping to use JDK 1.3 shinta
Re: FW: problem w/ ajp13 - if Tomcat is shutdown
What version of TC are you using? What version of Apache? I would look into the mod_jk docs -- I think this is the spec'd behavior (which, admittedly, is not great, but that makes it more of a feature request than a bug ;-). With ajp13, Apache opens up a persistent TCP/IP connection TC -- if TC restarts, I think that connection may just hang up and then timeout (since Apache doesn't know that TC has restarted). If anyone wants to work on this, you would have the undying thanks of the rest of the TC community -- having to restart Apache all the time bugs a *lot* of people. -Dan Shinta Tjio wrote: I'm having problem with mod_jk if ajp13 is used. The problem is often reproduced when Tomcat is shut down without Apache being shut down. When a request is fired through Apache as soon as Tomcat starts, I often get Internal Server Error. The mod_jk.log will have the following: [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13 [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 [jk_worker.c (127)]: wc_get_worker_for_name, done found a worker [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - jk_tcp_socket_recvfull failed [jk_ajp13_worker.c (619)]: Error reading request [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done If I hit reload multiple times, eventually Tomcat will serve the servlet fine. Did anyone see this problem before? Is there anyway around this? shinta -- Dan Milstein // [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I am using Tomcat 3.2.1, Apache 1.3.14, running on Solaris 2.8, Sun machines. After various attempts of debugging this, I have more information. 1. Even though I'm setting the worker's property cache_size to default (1), I'm finding there are up to 6 connections opened from Apache to Tomcat. I deduce this by looking at the mod_jk.conf and by doing netstat. I finally find out, this is so because my Apache is set to spawn minimum of 6 children and each of those children are making separate connections to Tomcat. This is very bad because, I ended up having to reload 6 times before Tomcat starts serving me the page again. Each time it uses a different Apache children that has defunct socket. So the more Apache children I have, the longer it takes me to recover from this problem. 2. It seems when Tomcat dies restarts, the send() called by ajp13's jk_tcp_socket_sendfull() does not get an error. But the recv() does get an error, with errno ECONNRESET. After which, the socket is properly closed. 3. When I shutdown Tomcat, those sockets that were opened between Apache/Tomcat showed up in state CLOSE_WAIT, and FIN_WAIT2. I think this is normally solved by calling the shutdown() API after closing the socket. However, this would have to be done from the Tomcat side in Ajp13ConnectionHandler.java. I can't find the corresponding method of Socket in Java. So.. based on all of these, the only fix I can think of putting is to make mod_jk retry the send() if recv() comes back with an error ECONNRESET. The retry should happen after the old socket is properly closed. Anyone wants to comment? shinta -Original Message- From: Dan Milstein [mailto:[EMAIL PROTECTED]] Sent: Tuesday, March 06, 2001 12:00 PM To: [EMAIL PROTECTED] Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown What version of TC are you using? What version of Apache? I would look into the mod_jk docs -- I think this is the spec'd behavior (which, admittedly, is not great, but that makes it more of a feature request than a bug ;-). With ajp13, Apache opens up a persistent TCP/IP connection TC -- if TC restarts, I think that connection may just hang up and then timeout (since Apache doesn't know that TC has restarted). If anyone wants to work on this, you would have the undying thanks of the rest of the TC community -- having to restart Apache all the time bugs a *lot* of people. -Dan Shinta Tjio wrote: I'm having problem with mod_jk if ajp13 is used. The problem is often reproduced when Tomcat is shut down without Apache being shut down. When a request is fired through Apache as soon as Tomcat starts, I often get Internal Server Error. The mod_jk.log will have the following: [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13 [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 [jk_worker.c (127)]: wc_get_worker_for_name, done found a worker [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - jk_tcp_socket_recvfull failed [jk_ajp13_worker.c (619)]: Error reading request [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done If I hit reload multiple times, eventually Tomcat will serve the servlet fine. Did anyone see this problem before? Is there anyway around this? shinta -- Dan Milstein // [EMAIL PROTECTED]
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
I was running recently with that exact same configuration but did not experience that problem. Can you show some config files. -Andy -Original Message- From: Shinta Tjio To: [EMAIL PROTECTED] Cc: 'Dan Milstein' Sent: 3/6/01 7:01 PM Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I am using Tomcat 3.2.1, Apache 1.3.14, running on Solaris 2.8, Sun machines. After various attempts of debugging this, I have more information. 1. Even though I'm setting the worker's property cache_size to default (1), I'm finding there are up to 6 connections opened from Apache to Tomcat. I deduce this by looking at the mod_jk.conf and by doing netstat. I finally find out, this is so because my Apache is set to spawn minimum of 6 children and each of those children are making separate connections to Tomcat. This is very bad because, I ended up having to reload 6 times before Tomcat starts serving me the page again. Each time it uses a different Apache children that has defunct socket. So the more Apache children I have, the longer it takes me to recover from this problem. 2. It seems when Tomcat dies restarts, the send() called by ajp13's jk_tcp_socket_sendfull() does not get an error. But the recv() does get an error, with errno ECONNRESET. After which, the socket is properly closed. 3. When I shutdown Tomcat, those sockets that were opened between Apache/Tomcat showed up in state CLOSE_WAIT, and FIN_WAIT2. I think this is normally solved by calling the shutdown() API after closing the socket. However, this would have to be done from the Tomcat side in Ajp13ConnectionHandler.java. I can't find the corresponding method of Socket in Java. So.. based on all of these, the only fix I can think of putting is to make mod_jk retry the send() if recv() comes back with an error ECONNRESET. The retry should happen after the old socket is properly closed. Anyone wants to comment? shinta -Original Message- From: Dan Milstein [ mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ] Sent: Tuesday, March 06, 2001 12:00 PM To: [EMAIL PROTECTED] Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown What version of TC are you using? What version of Apache? I would look into the mod_jk docs -- I think this is the spec'd behavior (which, admittedly, is not great, but that makes it more of a feature request than a bug ;-). With ajp13, Apache opens up a persistent TCP/IP connection TC -- if TC restarts, I think that connection may just hang up and then timeout (since Apache doesn't know that TC has restarted). If anyone wants to work on this, you would have the undying thanks of the rest of the TC community -- having to restart Apache all the time bugs a *lot* of people. -Dan Shinta Tjio wrote: I'm having problem with mod_jk if ajp13 is used. The problem is often reproduced when Tomcat is shut down without Apache being shut down. When a request is fired through Apache as soon as Tomcat starts, I often get Internal Server Error. The mod_jk.log will have the following: [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13 [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13 [jk_worker.c (127)]: wc_get_worker_for_name, done found a worker [jk_ajp13_worker.c (651)]: Into jk_worker_t::get_endpoint [jk_ajp13_worker.c (536)]: Into jk_endpoint_t::service [jk_ajp13.c (346)]: Into ajp13_marshal_into_msgb [jk_ajp13.c (480)]: ajp13_marshal_into_msgb - Done [jk_ajp13_worker.c (203)]: connection_tcp_get_message: Error - jk_tcp_socket_recvfull failed [jk_ajp13_worker.c (619)]: Error reading request [jk_ajp13_worker.c (489)]: Into jk_endpoint_t::done If I hit reload multiple times, eventually Tomcat will serve the servlet fine. Did anyone see this problem before? Is there anyway around this? shinta -- Dan Milstein // [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
RE: FW: problem w/ ajp13 - if Tomcat is shutdown
Title: RE: FW: problem w/ ajp13 - if Tomcat is shutdown Here's exactly what I did to reproduce the problem. Again, I'm running this on a Solaris 2.8 SunOS machine, using JDK1.2.2, Apache 1.3.14, Tomcat 3.2.1. 1. Start Apache 2. Start Tomcat 3. Start hitting Apache with multiple requests, such as /example/servlet/HelloWorldExample. Make sure there are some connections opened from Apache to Tomcat. To make sure, you can do netstat -a | grep Tomcat port where Tomcat port is the ajp13 port. It should show some socket in ESTABLISHED state. 4. While the connections are in the ESTABLISHED state (this should be the state because ajp13 reuse connections), shutdown Tomcat. Now you will notice the same netstat will show some sockets in FIN_WAIT2 and CLOSE_WAIT state. 5. Now restart Tomcat. 6. Repeat step 3. You will get Internal Server Error, up to the number of children Apache has. After that number, the page will be served. Attached is the server.xml, workers.properties, httpd.conf. My test servlet is called /mytest/servlet/testServlet but I think you can try it with any kind of servlet. I have been trying to code up the retry I mentioned below. I think I got it working. I just need to clean up the code some more. shinta -Original Message- From: oliver2, andy [mailto:[EMAIL PROTECTED]] Sent: Tuesday, March 06, 2001 6:33 PM To: 'Shinta Tjio '; '[EMAIL PROTECTED] ' Cc: ''Dan Milstein' ' Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I was running recently with that exact same configuration but did not experience that problem. Can you show some config files. -Andy -Original Message- From: Shinta Tjio To: [EMAIL PROTECTED] Cc: 'Dan Milstein' Sent: 3/6/01 7:01 PM Subject: RE: FW: problem w/ ajp13 - if Tomcat is shutdown I am using Tomcat 3.2.1, Apache 1.3.14, running on Solaris 2.8, Sun machines. After various attempts of debugging this, I have more information. 1. Even though I'm setting the worker's property cache_size to default (1), I'm finding there are up to 6 connections opened from Apache to Tomcat. I deduce this by looking at the mod_jk.conf and by doing netstat. I finally find out, this is so because my Apache is set to spawn minimum of 6 children and each of those children are making separate connections to Tomcat. This is very bad because, I ended up having to reload 6 times before Tomcat starts serving me the page again. Each time it uses a different Apache children that has defunct socket. So the more Apache children I have, the longer it takes me to recover from this problem. 2. It seems when Tomcat dies restarts, the send() called by ajp13's jk_tcp_socket_sendfull() does not get an error. But the recv() does get an error, with errno ECONNRESET. After which, the socket is properly closed. 3. When I shutdown Tomcat, those sockets that were opened between Apache/Tomcat showed up in state CLOSE_WAIT, and FIN_WAIT2. I think this is normally solved by calling the shutdown() API after closing the socket. However, this would have to be done from the Tomcat side in Ajp13ConnectionHandler.java. I can't find the corresponding method of Socket in Java. So.. based on all of these, the only fix I can think of putting is to make mod_jk retry the send() if recv() comes back with an error ECONNRESET. The retry should happen after the old socket is properly closed. Anyone wants to comment? shinta -Original Message- From: Dan Milstein [ mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ] Sent: Tuesday, March 06, 2001 12:00 PM To: [EMAIL PROTECTED] Subject: Re: FW: problem w/ ajp13 - if Tomcat is shutdown What version of TC are you using? What version of Apache? I would look into the mod_jk docs -- I think this is the spec'd behavior (which, admittedly, is not great, but that makes it more of a feature request than a bug ;-). With ajp13, Apache opens up a persistent TCP/IP connection TC -- if TC restarts, I think that connection may just hang up and then timeout (since Apache doesn't know that TC has restarted). If anyone wants to work on this, you would have the undying thanks of the rest of the TC community -- having to restart Apache all the time bugs a *lot* of people. -Dan Shinta Tjio wrote: I'm having problem with mod_jk if ajp13 is used. The problem is often reproduced when Tomcat is shut down without Apache being shut down. When a request is fired through Apache as soon as Tomcat starts, I often get Internal Server Error. The mod_jk.log will have the following: [jk_uri_worker_map.c (344)]: Into jk_uri_worker_map_t::map_uri_to_worker [jk_uri_worker_map.c (406)]: jk_uri_worker_map_t::map_uri_to_worker, Found a match ajp13 [jk_worker.c (123)]: Into wc_get_worker_for_name ajp13
