Re: [SPAM] RE: Application stops responding, jk worker in error state
On 19.06.2010 03:46, Neil Aggarwal wrote: Rainer: Hmmm, errno 11 is often EAGAIN. What platform are you using (OS, version)? I am using CentOS 5.5 freshly installed, not an upgrade. I tried that as a fix to this problem. I wiped the server clean and reinstalled everything from scratch. It was on CentOS 5.4 before and I had the same behavior. What is really strange is the JK Status page says -48 connections. It seems like that number should always be positive or zero but not negative. That should be unrelated (and yes: it's not OK). It seems to me if jk is confused about the number of backend connections, it may not be connecting correctly to Tomcat. That seems like it would cause the behavior I am experiencing. No, the total number of connections is shown only for convenience (monitoring etc.). The real process pool is local for each Apache process. there seems to be something wrong when tracking the total number via shared memory but that shouldn't influence in any way the usual functionality. It has only been added very recently. What *is* a problem is EAGAIN while reading from the socket. We don't expect that and handle it as an error, though strictly speaking it is only a temporary error condition. i'll see whether I can provide a patch. would you mind opening an issua in Bugzilla, attaching your log snippet and possibly your workers.properties (the property dump from the status worker is a little harder to digest, because it also shows all the defaults). You can also open a second issue concering the obvisouly wrong connection count shown in the status worker. Thanks and sory for the trouble Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Rainer: would you mind opening an issua in Bugzilla, attaching your log snippet and possibly your workers.properties I created bug 49468 in bugzilla, included my log snippet and attached the workers.properties file. You can also open a second issue concering the obvisouly wrong connection count shown in the status worker. I created bug 49469 in bugzilla and attached my screenshot showing the negative number of connections. Thanks and sory for the trouble Thank you for being so responsive! I hope we are able to fix the lock-up problem soon since this is a production system. I sincerely appreciate your help. Thanks, Neil -- Neil Aggarwal, (281)846-8957, http://UnmeteredVPS.net FREE trial: Tomcat 6 virtual server with unmetered bandwidth http://UnmeteredVPS.net/tomcat - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Rainer: If you are going to post part of your log file, please also do provide your mod_jk configuration and version information. I am using mod_jk 1.2.30 which I built from the source tarball. My application just locked up so I did a tail -f on the mod_jk log and made a request to it. Here is what generated in the log: [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=11) [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info] ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat failed (recoverable), (attempt=1) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=11) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info] ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat failed (recoverable), (attempt=2) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat) connecting to tomcat failed. [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info] jk_handler::mod_jk.c (2618): Service error=0 for worker=tomcat I took a look at the tomcat catalina.out and do not see any errors there. The CPU is empty and the RAM is available. I telnet to 127.0.0.1 port 8009 on the machine, I get a connection, so something is listening on the port. If I do a ps aux | grep java, I see two processes for tomcat which is normal. I checked the mysql log and do not see any errors. I did a show innodb status and do not see any deadlocks in the database. I am attaching the properties for my jk configuation and a screen shot of the JK Status in case it helps. What is really strange is the JK Status page says -48 connections. It seems like that number should always be positive or zero but not negative. I appreciate any help you can offer. Thanks, Neil -- Neil Aggarwal, (281)846-8957 FREE trial: Tomcat 6 VPS with unmetered bandwidth http://UnmeteredVPS.net/tomcat - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Rainer: I am attaching the properties for my jk configuation and a screen shot of the JK Status in case it helps. It looks like the mailing list software stripped the attachments from my email. Here is the list of jk properties: worker.server_name=tweb1.retcgroup.com worker.server_port=80 worker.time_datetime=20100618152329 worker.time_tz=CDT worker.time_unix=1276892609 worker.web_server=Apache/2.2.3 (CentOS) worker.jk_version=mod_jk/1.2.30 worker.ajp_count=1 worker.tomcat.list=tomcat worker.tomcat.type=ajp13 worker.tomcat.host=localhost worker.tomcat.port=8009 worker.tomcat.address=127.0.0.1:8009 worker.tomcat.connection_pool_timeout=600 worker.tomcat.ping_timeout=1 worker.tomcat.connect_timeout=0 worker.tomcat.prepost_timeout=0 worker.tomcat.reply_timeout=0 worker.tomcat.retries=2 worker.tomcat.connection_ping_interval=0 worker.tomcat.recovery_options=0 worker.tomcat.max_packet_size=8192 worker.tomcat.used=6043 worker.tomcat.errors=57 worker.tomcat.client_errors=36 worker.tomcat.reply_timeouts=0 worker.tomcat.transferred=1571835650 worker.tomcat.read=65081493100 worker.tomcat.busy=2 worker.tomcat.max_busy=21 worker.tomcat.connected=-39 worker.tomcat.map_count=2 worker.tomcat.last_reset_at=1276866423 worker.tomcat.last_reset_ago=26186 worker.tomcat.error_time_datetime=20100618151913 worker.tomcat.error_time_tz=CDT worker.tomcat.error_time_unix seconds=1276892353 worker.tomcat.error_time_ago seconds=256 worker.tomcat.map.1.server=tweb1.retcgroup.com worker.tomcat.map.1.uri=/thymeleweb/* worker.tomcat.map.1.type=Wildchar worker.tomcat.map.1.source=JkMount worker.tomcat.map.1.reply_timeout=-1 worker.tomcat.map.1.fail_on_status= worker.tomcat.map.1.active= worker.tomcat.map.1.disabled= worker.tomcat.map.1.stopped= worker.tomcat.map.1.use_server_errors=0 worker.tomcat.map.2.server=tweb1.retcgroup.com [_default_:443] worker.tomcat.map.2.uri=/thymeleweb/* worker.tomcat.map.2.type=Wildchar worker.tomcat.map.2.source=JkMount worker.tomcat.map.2.reply_timeout=-1 worker.tomcat.map.2.fail_on_status= worker.tomcat.map.2.active= worker.tomcat.map.2.disabled= worker.tomcat.map.2.stopped= worker.tomcat.map.2.use_server_errors=0 worker.result.type=OK worker.result.message=Action finished The screenshot of my jk status page is at: http://tweb1.retcgroup.com/screenshot.gif Thanks, Neil -- Neil Aggarwal, (281)846-8957, http://UnmeteredVPS.net FREE trial: Tomcat 6 VPS with unmetered bandwidth http://UnmeteredVPS.net/tomcat - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Application stops responding, jk worker in error state
On 18.06.2010 22:44, Neil Aggarwal wrote: Rainer: If you are going to post part of your log file, please also do provide your mod_jk configuration and version information. I am using mod_jk 1.2.30 which I built from the source tarball. My application just locked up so I did a tail -f on the mod_jk log and made a request to it. Here is what generated in the log: [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=11) Hmmm, errno 11 is often EAGAIN. What platform are you using (OS, version)? [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jun 18 15:30:17.614 2010] [394:47652193829632] [info] ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat failed (recoverable), (attempt=1) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1230): (tomcat) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=11) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat) Tomcat is down or refused connection. No response has been sent to the client (yet) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info] ajp_service::jk_ajp_common.c (2540): (tomcat) sending request to tomcat failed (recoverable), (attempt=2) [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat) connecting to tomcat failed. [Fri Jun 18 15:31:19.711 2010] [394:47652193829632] [info] jk_handler::mod_jk.c (2618): Service error=0 for worker=tomcat I took a look at the tomcat catalina.out and do not see any errors there. The CPU is empty and the RAM is available. I telnet to 127.0.0.1 port 8009 on the machine, I get a connection, so something is listening on the port. If I do a ps aux | grep java, I see two processes for tomcat which is normal. I checked the mysql log and do not see any errors. I did a show innodb status and do not see any deadlocks in the database. I am attaching the properties for my jk configuation and a screen shot of the JK Status in case it helps. What is really strange is the JK Status page says -48 connections. It seems like that number should always be positive or zero but not negative. That should be unrelated (and yes: it's not OK). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Application stops responding, jk worker in error state
Rainer: Hmmm, errno 11 is often EAGAIN. What platform are you using (OS, version)? I am using CentOS 5.5 freshly installed, not an upgrade. I tried that as a fix to this problem. I wiped the server clean and reinstalled everything from scratch. It was on CentOS 5.4 before and I had the same behavior. What is really strange is the JK Status page says -48 connections. It seems like that number should always be positive or zero but not negative. That should be unrelated (and yes: it's not OK). It seems to me if jk is confused about the number of backend connections, it may not be connecting correctly to Tomcat. That seems like it would cause the behavior I am experiencing. Thanks, Neil -- Neil Aggarwal, (281)846-8957 FREE trial: Virtualmin VPS with unmetered bandwidth http://UnmeteredVPS.net/virtualmin - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Application stops responding, jk worker in error state
Hello: I have Tomcat 6.0.26 running behind Apache and am using the JK Connector to communicate between them. My application stops responding on occasion. I added the jk status page to the web server and checked it when the application becomes non-responsive. The tomcat worker has this info: Worker Status for tomcat Type Hostname Address:Port Connection Pool Timeout Connect Timeout Prepost Timeout Reply Timeout Retries Recovery Options Max Packet Size ajp13 localhost 127.0.0.1:8009 600 0 0 0 2 0 State Acc Err CE RE Wr Rd Busy Max LR LE ERR 6 (0/sec) 2 0 0 4.8K (7 /sec) 12K (18 /sec) 0 2 663 Fri, 18 Jun 2010 00:06:17 CDT Note the state is ERR. Does this mean the worker crashed or does it mean my application is locked up? Any ideas how to diagnose this? Thanks, Neil -- Neil Aggarwal, (281)846-8957 FREE trial: Virtualmin VPS with unmetered bandwidth http://UnmeteredVPS.net/virtualmin - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
