Re: unknown state in VisualVM
In case this would help further, I also see this in my debug mod_jk log: [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to ( 127.0.0.1:8011) (errno=111) [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [error] ajp_send_request::jk_ajp_common.c (1630): (worker3) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] ajp_service::jk_ajp_common.c (2623): (worker3) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1) [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] service::jk_lb_worker.c (1558): All tomcat instances are busy or in error state [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [error] service::jk_lb_worker.c (1563): All tomcat instances failed, no more workers left [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] jk_handler::mod_jk.c (2788): Service error=0 for worker=lbworker The port used for worker3 should definitely have been listening. Is this normal? The application is under load, 1500 jmeter users. Thanks, Charles On Mon, Oct 29, 2012 at 11:48 AM, Charles Richard charle...@thelearningbar.com wrote: Load test users from Jmeter. Cheers, Charles On Mon, Oct 29, 2012 at 11:25 AM, André Warnier a...@ice-sa.com wrote: Charles Richard wrote: I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would Just a side question : where are these users ? are these processes running on the same machine ? --**--**- To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.orgusers-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: unknown state in VisualVM
Actually, never mind, wrong timestamp, apache and mod_jk are seemingly down during load with no errors in Apache or mod_jk log. Thanks, Charles On Wed, Oct 31, 2012 at 3:38 PM, Charles Richard charle...@thelearningbar.com wrote: In case this would help further, I also see this in my debug mod_jk log: [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] ajp_connect_to_endpoint::jk_ajp_common.c (995): Failed opening socket to ( 127.0.0.1:8011) (errno=111) [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [error] ajp_send_request::jk_ajp_common.c (1630): (worker3) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=111) [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] ajp_service::jk_ajp_common.c (2623): (worker3) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1) [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] service::jk_lb_worker.c (1558): All tomcat instances are busy or in error state [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [error] service::jk_lb_worker.c (1563): All tomcat instances failed, no more workers left [Wed Oct 31 11:33:16 2012] [4110:47377141191536] [info] jk_handler::mod_jk.c (2788): Service error=0 for worker=lbworker The port used for worker3 should definitely have been listening. Is this normal? The application is under load, 1500 jmeter users. Thanks, Charles On Mon, Oct 29, 2012 at 11:48 AM, Charles Richard charle...@thelearningbar.com wrote: Load test users from Jmeter. Cheers, Charles On Mon, Oct 29, 2012 at 11:25 AM, André Warnier a...@ice-sa.com wrote: Charles Richard wrote: I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would Just a side question : where are these users ? are these processes running on the same machine ? --**--** - To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.orgusers-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
unknown state in VisualVM
Hi, I had a thread a while back but my configuration has changed and so has my issue, so hence a new thread. My config: CentOS 5.8 Apache 2.23 3 tomcat instances (6.0.30) with each ajp_port with a connectionTimeout setting of 18 mod_jk 1.2.37 workers.properties: # Define 1 real worker using ajp13 worker.list=worker1, worker2, worker3, lbworker # Set properties for worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_timeout=180 worker.worker1.lbfactor=1 worker.worker1.ping_mode=A worker.worker1.socket_keepalive=true # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8010 worker.worker2.connection_pool_timeout=180 worker.worker2.lbfactor=1 worker.worker2.ping_mode=A worker.worker2.socket_keepalive=true # Set properties for worker3 (ajp13) worker.worker3.type=ajp13 worker.worker3.host=localhost worker.worker3.port=8011 worker.worker3.connection_pool_timeout=180 worker.worker3.lbfactor=1 worker.worker3.ping_mode=A worker.worker3.socket_keepalive=true worker.lbworker.type=lb worker.lbworker.balance_workers=worker1, worker2, worker3 worker.lbworker.sticky_session=True I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would turn white in a off/unknown state (can send a snapshot of this on demand). Since then, I added the ping_mode and socket_keepalive setting in my workers.properties file but that didn't seem to make a difference. All i see in my mod_jk log is this: [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) Any idea what could be going on? Thanks, Charles
Re: unknown state in VisualVM
To add to this, apache seems to be hung up as well as it can't even serve regular non mod_jk content but there's no error messages. Cheers, Charles On Mon, Oct 29, 2012 at 9:37 AM, Charles Richard charle...@thelearningbar.com wrote: Hi, I had a thread a while back but my configuration has changed and so has my issue, so hence a new thread. My config: CentOS 5.8 Apache 2.23 3 tomcat instances (6.0.30) with each ajp_port with a connectionTimeout setting of 18 mod_jk 1.2.37 workers.properties: # Define 1 real worker using ajp13 worker.list=worker1, worker2, worker3, lbworker # Set properties for worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_timeout=180 worker.worker1.lbfactor=1 worker.worker1.ping_mode=A worker.worker1.socket_keepalive=true # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8010 worker.worker2.connection_pool_timeout=180 worker.worker2.lbfactor=1 worker.worker2.ping_mode=A worker.worker2.socket_keepalive=true # Set properties for worker3 (ajp13) worker.worker3.type=ajp13 worker.worker3.host=localhost worker.worker3.port=8011 worker.worker3.connection_pool_timeout=180 worker.worker3.lbfactor=1 worker.worker3.ping_mode=A worker.worker3.socket_keepalive=true worker.lbworker.type=lb worker.lbworker.balance_workers=worker1, worker2, worker3 worker.lbworker.sticky_session=True I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would turn white in a off/unknown state (can send a snapshot of this on demand). Since then, I added the ping_mode and socket_keepalive setting in my workers.properties file but that didn't seem to make a difference. All i see in my mod_jk log is this: [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) Any idea what could be going on? Thanks, Charles
Re: unknown state in VisualVM
On Oct 29, 2012, at 8:37 AM, Charles Richard wrote: Hi, I had a thread a while back but my configuration has changed and so has my issue, so hence a new thread. My config: CentOS 5.8 Apache 2.23 3 tomcat instances (6.0.30) with each ajp_port with a connectionTimeout setting of 18 mod_jk 1.2.37 workers.properties: # Define 1 real worker using ajp13 worker.list=worker1, worker2, worker3, lbworker # Set properties for worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_timeout=180 worker.worker1.lbfactor=1 worker.worker1.ping_mode=A worker.worker1.socket_keepalive=true # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8010 worker.worker2.connection_pool_timeout=180 worker.worker2.lbfactor=1 worker.worker2.ping_mode=A worker.worker2.socket_keepalive=true # Set properties for worker3 (ajp13) worker.worker3.type=ajp13 worker.worker3.host=localhost worker.worker3.port=8011 worker.worker3.connection_pool_timeout=180 worker.worker3.lbfactor=1 worker.worker3.ping_mode=A worker.worker3.socket_keepalive=true worker.lbworker.type=lb worker.lbworker.balance_workers=worker1, worker2, worker3 worker.lbworker.sticky_session=True I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would turn white in a off/unknown state (can send a snapshot of this on demand). Since then, I added the ping_mode and socket_keepalive setting in my workers.properties file but that didn't seem to make a difference. All i see in my mod_jk log is this: [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) Any idea what could be going on? Do you have a firewall between Apache HTTPD and your Tomcat instances? Dan Thanks, Charles - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: unknown state in VisualVM
No firewall, all on one server. Right now, I'm thinking I'm either hitting some type of tcp connection maximum or something is going on with Apache or mod_jk. If I just restart Apache, i can connect to Tomcat again which tells me Tomcat wasn't down. Cheers, Charles On Mon, Oct 29, 2012 at 10:39 AM, Daniel Mikusa dmik...@vmware.com wrote: On Oct 29, 2012, at 8:37 AM, Charles Richard wrote: Hi, I had a thread a while back but my configuration has changed and so has my issue, so hence a new thread. My config: CentOS 5.8 Apache 2.23 3 tomcat instances (6.0.30) with each ajp_port with a connectionTimeout setting of 18 mod_jk 1.2.37 workers.properties: # Define 1 real worker using ajp13 worker.list=worker1, worker2, worker3, lbworker # Set properties for worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_timeout=180 worker.worker1.lbfactor=1 worker.worker1.ping_mode=A worker.worker1.socket_keepalive=true # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8010 worker.worker2.connection_pool_timeout=180 worker.worker2.lbfactor=1 worker.worker2.ping_mode=A worker.worker2.socket_keepalive=true # Set properties for worker3 (ajp13) worker.worker3.type=ajp13 worker.worker3.host=localhost worker.worker3.port=8011 worker.worker3.connection_pool_timeout=180 worker.worker3.lbfactor=1 worker.worker3.ping_mode=A worker.worker3.socket_keepalive=true worker.lbworker.type=lb worker.lbworker.balance_workers=worker1, worker2, worker3 worker.lbworker.sticky_session=True I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would turn white in a off/unknown state (can send a snapshot of this on demand). Since then, I added the ping_mode and socket_keepalive setting in my workers.properties file but that didn't seem to make a difference. All i see in my mod_jk log is this: [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) Any idea what could be going on? Do you have a firewall between Apache HTTPD and your Tomcat instances? Dan Thanks, Charles - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: unknown state in VisualVM
On Oct 29, 2012, at 9:41 AM, Charles Richard wrote: No firewall, all on one server. Right now, I'm thinking I'm either hitting some type of tcp connection maximum or something is going on with Apache or mod_jk. Think you might be on the right track here. I just google'd errno=24 and it came up with Too many open files. Dan If I just restart Apache, i can connect to Tomcat again which tells me Tomcat wasn't down. Cheers, Charles On Mon, Oct 29, 2012 at 10:39 AM, Daniel Mikusa dmik...@vmware.com wrote: On Oct 29, 2012, at 8:37 AM, Charles Richard wrote: Hi, I had a thread a while back but my configuration has changed and so has my issue, so hence a new thread. My config: CentOS 5.8 Apache 2.23 3 tomcat instances (6.0.30) with each ajp_port with a connectionTimeout setting of 18 mod_jk 1.2.37 workers.properties: # Define 1 real worker using ajp13 worker.list=worker1, worker2, worker3, lbworker # Set properties for worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_timeout=180 worker.worker1.lbfactor=1 worker.worker1.ping_mode=A worker.worker1.socket_keepalive=true # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8010 worker.worker2.connection_pool_timeout=180 worker.worker2.lbfactor=1 worker.worker2.ping_mode=A worker.worker2.socket_keepalive=true # Set properties for worker3 (ajp13) worker.worker3.type=ajp13 worker.worker3.host=localhost worker.worker3.port=8011 worker.worker3.connection_pool_timeout=180 worker.worker3.lbfactor=1 worker.worker3.ping_mode=A worker.worker3.socket_keepalive=true worker.lbworker.type=lb worker.lbworker.balance_workers=worker1, worker2, worker3 worker.lbworker.sticky_session=True I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would turn white in a off/unknown state (can send a snapshot of this on demand). Since then, I added the ping_mode and socket_keepalive setting in my workers.properties file but that didn't seem to make a difference. All i see in my mod_jk log is this: [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) Any idea what could be going on? Do you have a firewall between Apache HTTPD and your Tomcat instances? Dan Thanks, Charles - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: unknown state in VisualVM
Looks like you were right Daniel, i increased the open file limit from 32768 to 128768 and the same problem is persisting but there's no logging in the mod_jk.log anymore which seems to indicate increasing the file limit made a difference. Thanks for the help! Charles On Mon, Oct 29, 2012 at 10:48 AM, Daniel Mikusa dmik...@vmware.com wrote: On Oct 29, 2012, at 9:41 AM, Charles Richard wrote: No firewall, all on one server. Right now, I'm thinking I'm either hitting some type of tcp connection maximum or something is going on with Apache or mod_jk. Think you might be on the right track here. I just google'd errno=24 and it came up with Too many open files. Dan If I just restart Apache, i can connect to Tomcat again which tells me Tomcat wasn't down. Cheers, Charles On Mon, Oct 29, 2012 at 10:39 AM, Daniel Mikusa dmik...@vmware.com wrote: On Oct 29, 2012, at 8:37 AM, Charles Richard wrote: Hi, I had a thread a while back but my configuration has changed and so has my issue, so hence a new thread. My config: CentOS 5.8 Apache 2.23 3 tomcat instances (6.0.30) with each ajp_port with a connectionTimeout setting of 18 mod_jk 1.2.37 workers.properties: # Define 1 real worker using ajp13 worker.list=worker1, worker2, worker3, lbworker # Set properties for worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.connection_pool_timeout=180 worker.worker1.lbfactor=1 worker.worker1.ping_mode=A worker.worker1.socket_keepalive=true # Set properties for worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8010 worker.worker2.connection_pool_timeout=180 worker.worker2.lbfactor=1 worker.worker2.ping_mode=A worker.worker2.socket_keepalive=true # Set properties for worker3 (ajp13) worker.worker3.type=ajp13 worker.worker3.host=localhost worker.worker3.port=8011 worker.worker3.connection_pool_timeout=180 worker.worker3.lbfactor=1 worker.worker3.ping_mode=A worker.worker3.socket_keepalive=true worker.lbworker.type=lb worker.lbworker.balance_workers=worker1, worker2, worker3 worker.lbworker.sticky_session=True I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would turn white in a off/unknown state (can send a snapshot of this on demand). Since then, I added the ping_mode and socket_keepalive setting in my workers.properties file but that didn't seem to make a difference. All i see in my mod_jk log is this: [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:16:56 2012] [12205:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_handle_cping_cpong::jk_ajp_common.c (913): timeout in reply cpong [Mon Oct 29 09:18:36 2012] [10542:47535048426352] [info] ajp_maintain::jk_ajp_common.c (3289): (worker2) failed sending request, socket -1 keepalive cping/cpong failure (errno=24) Any idea what could be going on? Do you have a firewall between Apache HTTPD and your Tomcat instances? Dan Thanks, Charles - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: unknown state in VisualVM
Charles Richard wrote: I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would Just a side question : where are these users ? are these processes running on the same machine ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: unknown state in VisualVM
Load test users from Jmeter. Cheers, Charles On Mon, Oct 29, 2012 at 11:25 AM, André Warnier a...@ice-sa.com wrote: Charles Richard wrote: I'm doing some performance tuning on our application and noticed a couple of days ago that which a load of 1500 users, my threads in VisualVM would Just a side question : where are these users ? are these processes running on the same machine ? --**--**- To unsubscribe, e-mail: users-unsubscribe@tomcat.**apache.orgusers-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
