Re: 2 second delays in mod_jk while maintaining workers

2010-09-15 Thread John Baker
Hello, I've spent a week looking into mod_jk, Apache, etc., and the problem appears to be due to a bug in the AJP connector within JBoss: https://jira.jboss.org/browse/JBPAPP-366 This bug is not fixed in the latest release of JBoss 4.2.3, which is rather disappointing. After downloading the

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread John Baker
Thanks for the feedback. Can you tell me why this if statement exists: if (poll(fds, 1, timeout) 0) { ... } else break; It appears to be at fault. John - To unsubscribe,

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread Mladen Turk
On 09/09/2010 02:09 PM, John Baker wrote: Thanks for the feedback. Can you tell me why this if statement exists: if (poll(fds, 1, timeout) 0) { ... } else break; It appears to be at fault. poll returns positive number if there is

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread Jim Jagielski
On Sep 9, 2010, at 8:34 AM, Mladen Turk wrote: On 09/09/2010 02:09 PM, John Baker wrote: Thanks for the feedback. Can you tell me why this if statement exists: if (poll(fds, 1, timeout) 0) { ... } else break; It appears to be at

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread John Baker
On Thursday 09 September 2010 13:59:50 you wrote: If that doesn't help, it's obvious the Tomcat doesn't close the socket, so should be investigated why. Like said before, either the Tomcat doesn't respond to shutdown or the shutdown's FIN packet isn't send to the Tomcat or back to

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread Mladen Turk
On 09/09/2010 03:22 PM, John Baker wrote: On Thursday 09 September 2010 13:59:50 you wrote: If that doesn't help, it's obvious the Tomcat doesn't close the socket, so should be investigated why. Like said before, either the Tomcat doesn't respond to shutdown or the shutdown's FIN packet isn't

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread Mladen Turk
On 09/09/2010 03:22 PM, John Baker wrote: Do you fancy putting that change into the next release of mod_jk? BTW, do you fancy opening BZ report fill in what we tried so far and the actual solution, so we can track that down for the next releases and patches? Regards -- ^TM

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread John Baker
On Thursday 09 September 2010 15:45:44 you wrote: Nice. I spoke too soon. I'm now trying to figure out how to print out the IP address of the socket (I don't really do C) so I can log the socket that caused the poll to timeout, and compare with a tcpdump.

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread John Baker
I would be happy to share all my evidence and write a report once we get to the bottom of this problem. Any hints on printing out the socket IP (i.e. of Tomcat)? On Thursday 09 September 2010 15:47:33 you wrote: On 09/09/2010 03:22 PM, John Baker wrote: Do you fancy putting that change

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread John Baker
Interestingly, some of our JBoss instances are showing a large number ajp threads that seem to be in keep alive mode but are well beyond the connectionTimeout defined in server.xml (which is set to 9): Max threads: 40 Current thread count: 40 Current thread busy: 40 Max processing time:

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread Mladen Turk
On 09/09/2010 04:48 PM, John Baker wrote: On Thursday 09 September 2010 15:45:44 you wrote: Nice. I spoke too soon. I'm now trying to figure out how to print out the IP address of the socket (I don't really do C) so I can log the socket that caused the poll to timeout, and compare with a

Re: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread John Baker
On Thursday 09 September 2010 16:08:04 you wrote: On 09/09/2010 04:48 PM, John Baker wrote: On Thursday 09 September 2010 15:45:44 you wrote: Nice. I spoke too soon. I'm now trying to figure out how to print out the IP address of the socket (I don't really do C) so I can log the

RE: 2 second delays in mod_jk while maintaining workers

2010-09-09 Thread Caldarale, Charles R
From: John Baker [mailto:jba...@javasystemsolutions.com] Subject: Re: 2 second delays in mod_jk while maintaining workers some of our JBoss instances are showing a large number ajp threads that seem to be in keep alive mode but are well beyond the connectionTimeout defined in server.xml

Re: 2 second delays in mod_jk while maintaining workers

2010-09-08 Thread John Baker
Hello We've just noticed that the maintenance mode operates on all workers, so having one worker run maintenance for the rest is making tracing the problem difficult. Reading down the logs, we can see it finds a worker and iterates through workers perforing maintenance. Is this correct? Moving

Re: 2 second delays in mod_jk while maintaining workers

2010-09-08 Thread John Baker
Hello, I don't thikn the shutdown call is to blame - I think it's the large pile of code below. i.e. everything below the if (shutdown(..)) below. The question is, what does it all do and does it actually work? It appears to be the 'drain' code, but given it often results in this message:

Re: 2 second delays in mod_jk while maintaining workers

2010-09-08 Thread Mladen Turk
On 09/08/2010 05:08 PM, John Baker wrote: Hello, I don't thikn the shutdown call is to blame - I think it's the large pile of code below. i.e. everything below the if (shutdown(..)) below. The question is, what does it all do and does it actually work? It appears to be the 'drain' code, but

Re: 2 second delays in mod_jk while maintaining workers

2010-09-08 Thread Timothy Grocott
please can you remove me off the mailing list many thanks On Wed, Sep 8, 2010 at 4:29 PM, Mladen Turk mt...@apache.org wrote: On 09/08/2010 05:08 PM, John Baker wrote: Hello, I don't thikn the shutdown call is to blame - I think it's the large pile of code below. i.e. everything below

Re: 2 second delays in mod_jk while maintaining workers

2010-09-08 Thread John Baker
On 09/08/2010 05:08 PM, John Baker wrote: The code *is* required. It is used when the client disconnects while the backend still has some data in the AJP buffer. Drain is needed to read that excess data. Why does it always report 0 bytes read? If you can compile mod_jk and test, try

Re: 2 second delays in mod_jk while maintaining workers

2010-09-08 Thread John Baker
-- From: John Baker To: Tomcat Users List ReplyTo: Tomcat Users List Subject: Re: 2 second delays in mod_jk while maintaining workers Sent: 8 Sep 2010 16:41 On 09/08/2010 05:08 PM, John Baker wrote: The code *is* required. It is used when the client disconnects while the backend still has some

Re: 2 second delays in mod_jk while maintaining workers

2010-09-08 Thread Mladen Turk
On 09/08/2010 05:51 PM, John Baker wrote: I think it would be helpful if you could walk us through the code. It is very simple actually. 1. shutdown write end of our connection to Tomcat This should cause the soket.read() in Tomcat to throw exception which is used to close the socket

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
On Tuesday 07 September 2010 06:28:33 you wrote: On 09/06/2010 11:59 PM, John Baker wrote: What's jk_maintain? Function that maintains the workers (closes excess connections inactive for a long time) Anyhow, like Rainer said, if that's the case you should have something like this in

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Mladen Turk
On 09/07/2010 09:05 AM, John Baker wrote: On Tuesday 07 September 2010 06:28:33 you wrote: On 09/06/2010 11:59 PM, John Baker wrote: What's jk_maintain? Function that maintains the workers (closes excess connections inactive for a long time) Anyhow, like Rainer said, if that's the case you

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Rainer Jung
On 07.09.2010 09:05, John Baker wrote: On Tuesday 07 September 2010 06:28:33 you wrote: On 09/06/2010 11:59 PM, John Baker wrote: What's jk_maintain? Function that maintains the workers (closes excess connections inactive for a long time) Anyhow, like Rainer said, if that's the case you

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
On Tuesday 07 September 2010 08:59:27 you wrote: It means that socket shutdown failed. Do you have firewall between mod_jk and JBoss or some non-standard network driver (e.g running under some VM)? We are using VMs but there should be no firewall. I should probably re-iterate that the

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Mladen Turk
On 09/07/2010 10:11 AM, John Baker wrote: On Tuesday 07 September 2010 08:59:27 you wrote: It means that socket shutdown failed. Do you have firewall between mod_jk and JBoss or some non-standard network driver (e.g running under some VM)? We are using VMs but there should be no firewall. I

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
Rainer, Is this acceptable? I am using a tail and an egrep to match the various statements you wish to see. if it's missing anything, plesae let me know what to add to the grep. [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] find_match::jk_uri_worker_map.c (850): Attempting

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
TM, On Tuesday 07 September 2010 09:47:17 you wrote: If you have enough resources, try to disable connectionTimeout on AJP connector and see weather the same will happen again. I've removed the connectionTimeout attribute and the problem persists - I can see the lingering bytes in 2 sec

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Rainer Jung
On 07.09.2010 11:36, John Baker wrote: Rainer, Is this acceptable? I am using a tail and an egrep to match the various statements you wish to see. if it's missing anything, plesae let me know what to add to the grep. Yes, that's better! [Tue Sep 07 10:20:20.617 2010]

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Mladen Turk
On 09/07/2010 11:53 AM, John Baker wrote: TM, On Tuesday 07 September 2010 09:47:17 you wrote: If you have enough resources, try to disable connectionTimeout on AJP connector and see weather the same will happen again. I've removed the connectionTimeout attribute and the problem persists - I

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
Rainer, On Tuesday 07 September 2010 11:09:46 you wrote: [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768] [debug] find_match::jk_uri_worker_map.c (850): Attempting to map context URI '/*=lb-jboss51-integration' source 'JkMount' [Tue Sep 07 10:20:20.617 2010] [18806:46962404156768]

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
On Tuesday 07 September 2010 11:13:07 you wrote: It's obvious that shutdown(socket, SHUT_WR) poll(socket, 2 seconds) close(socket) caused poll call to time out, meaning that the JBoss side didn't respond to the shutdown(socket, SHUT_WR) call by closing it's side of the connection.

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Mladen Turk
On 09/07/2010 12:09 PM, Rainer Jung wrote: Are we sure, that those log lines are correct? This was generated by mod_jk 1.2.30 without any code changes, right? The other option would be to set JkWatchdogInterval 60 In that case wc_maintain would happen in separate thread without observable

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
Rainer, On Tuesday 07 September 2010 11:09:46 you wrote: I don't like socket_timeout ... worker.basic.socket_timeout=90 but I would like socket_connect_timeout. The next two are possibly a bit short, because if the backend e.g. does a Java Garbage Collection which miht take longer

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Rainer Jung
On 07.09.2010 12:29, Mladen Turk wrote: On 09/07/2010 12:09 PM, Rainer Jung wrote: Are we sure, that those log lines are correct? This was generated by mod_jk 1.2.30 without any code changes, right? The other option would be to set JkWatchdogInterval 60 In that case wc_maintain would happen

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Mladen Turk
On 09/07/2010 12:20 PM, John Baker wrote: On Tuesday 07 September 2010 11:13:07 you wrote: It's obvious that shutdown(socket, SHUT_WR) poll(socket, 2 seconds) close(socket) caused poll call to time out, meaning that the JBoss side didn't respond to the shutdown(socket, SHUT_WR) call by

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Rainer Jung
On 07.09.2010 12:32, John Baker wrote: Rainer, On Tuesday 07 September 2010 11:09:46 you wrote: I don't like socket_timeout ... worker.basic.socket_timeout=90 but I would like socket_connect_timeout. The next two are possibly a bit short, because if the backend e.g. does a Java Garbage

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Rainer Jung
On 07.09.2010 12:40, Rainer Jung wrote: On 07.09.2010 12:32, John Baker wrote: Rainer, On Tuesday 07 September 2010 11:09:46 you wrote: I don't like socket_timeout ... worker.basic.socket_timeout=90 but I would like socket_connect_timeout. The next two are possibly a bit short, because

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread John Baker
Where can I find documentation on JkWatchdog? On Tuesday 07 September 2010 13:51:23 you wrote: - As a workaround: using a JkWatchdog moves the maintain into a separate thread. But during the socket closing a lock is held, which blocks other threads from accessing the same worker.

Re: 2 second delays in mod_jk while maintaining workers

2010-09-07 Thread Rainer Jung
On 07.09.2010 19:46, John Baker wrote: Where can I find documentation on JkWatchdog? General documntation on mod_jk: http://tomcat.apache.org/connectors-doc/ JK directives: http://tomcat.apache.org/connectors-doc/reference/apache.html workers.properties:

2 second delays in mod_jk while maintaining workers

2010-09-06 Thread John Baker
Hello, I've discovered what appears to be a bug in mod_jk 1.2.27 and have also tried 1.2.30 without success. I'm using Apache 2.2.3 (on Redhat EL 5.4). The problem occurs after previous successful activity and causes a delay in what looks like socket handling. I noticed bug was resolved in

Re: 2 second delays in mod_jk while maintaining workers

2010-09-06 Thread Mladen Turk
On 09/06/2010 04:16 PM, John Baker wrote: I've set the Jk logging to trace and you can see the debug statements and the 2s delays: Do you use NFS share by any chance to store the mod_jk log directory data? What happens if you set worker.lb-jboss51-integration.lock=O Regards -- ^TM

Re: 2 second delays in mod_jk while maintaining workers

2010-09-06 Thread John Baker
On Monday 06 September 2010 18:56:20 you wrote: On 09/06/2010 04:16 PM, John Baker wrote: I've set the Jk logging to trace and you can see the debug statements and the 2s delays: Do you use NFS share by any chance to store the mod_jk log directory data? Nope. All log files are

Re: 2 second delays in mod_jk while maintaining workers

2010-09-06 Thread Rainer Jung
On 06.09.2010 19:56, Mladen Turk wrote: On 09/06/2010 04:16 PM, John Baker wrote: I've set the Jk logging to trace and you can see the debug statements and the 2s delays: Do you use NFS share by any chance to store the mod_jk log directory data? I didn't look at the code now, but the 2

Re: 2 second delays in mod_jk while maintaining workers

2010-09-06 Thread John Baker
On Monday 06 September 2010 22:57:21 you wrote: I didn't look at the code now, but the 2 seconds remind me of the connection draining during socket shutdown, which could be related to jk_maintain? What's jk_maintain? - To

Re: 2 second delays in mod_jk while maintaining workers

2010-09-06 Thread Mladen Turk
On 09/06/2010 11:59 PM, John Baker wrote: On Monday 06 September 2010 22:57:21 you wrote: I didn't look at the code now, but the 2 seconds remind me of the connection draining during socket shutdown, which could be related to jk_maintain? What's jk_maintain? Function that maintains the