Re: ajp_ilink_receive error - please advise

2012-09-21 Thread Daniel Mikusa 
On Sep 20, 2012, at 6:12 PM, Django Radonich-Camp wrote:

 thank you for the replies chris and dan.
 
 chris:
 1.  upgrading apache is possible, but not easy (puppet manages the install).
 2.  unfortunately i have not figured out how to reproduce the problem.  we
 did extensive load testing on the system prior to launch and not once did
 we see this problem...
 
 dan:
 1.  good idea.  do you know a good reference for setting this up?

It is just a matter of adding the JVM options in setenv.sh.

I'll typically use something like this...

-XX:+PrintGC -Xloggc:$CATALINA_BASE/logs/gc.log -XX:+PrintGCDetails 
-XX:+PrintGCTimeStamps

You can adjust the options to fit your needs.  If you need more detail, I would 
suggest google'ing java garbage collection log or the individual option.

 2.  also, what is the process for capturing thread dumps?  this will be
 difficult however because the incidents are random and so far, mostly
 ocurring in the middle of the night.

There are a few ways to do this.  I find running kill -3 pid to be the 
easiest.  You could also use jstack pid

Dan


 
 thanks again for the help.
 
 
 On Thu, Sep 20, 2012 at 5:35 AM, Daniel Mikusa dmik...@vmware.com wrote:
 
 On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote:
 
 hello.
 
 we are running an application on tomcat and experiencing intermittent
 periods where the application is non-responsive and thus non-functional.
 the general set up is apache and tomcat, with mod_proxy_ajp as the
 connector (specific details and configs below).
 
 during these events, the primary apache error log shows the following:
 1. (104)Connection reset by peer: ajp_ilink_receive() can't receive
 header
 2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
 receive header
 
 it looks like error #1 repeats for a while and then error #2 repeats for
 a
 while.
 
 at the same time in the application specific apache error log (as
 specified
 in the vhost set up for app) we see the following errors repeated (though
 in mixed order from below):
 3. [error] ajp_read_header: ajp_ilink_receive failed
 4. [error] (120006)APR does not understand this error code: proxy: read
 response failed from (null) (localhost)
 5. [error] (120006)APR does not understand this error code: proxy: read
 response failed from 127.0.0.1:8009 (localhost)
 6. [error] [client X.X.X.X] proxy: error processing end
 
 and occasionally:
 7. [error] proxy: read zero bytes, expecting 464 bytes
 
 the catalina.out log registers nothing during the time period the
 application is unresponsive.
 
 a couple of other things to note:
 - these events are coming under light to no load as far as i can tell.
 - these events last from about 5 to 30 minutes and then everything works
 again as expected with no manual intervention.
 - the time of day of for the events is not consistent.
 - these events initially occured rarely, but over the last month have
 ramped up to daily.
 
 any suggestions on how to fix or further troubleshoot this problem? and
 thanks in advance for the help.
 
 Here are a couple additional troubleshooting steps you can take on your
 Tomcat instances...
 
 1.) Enable garbage collection logging.  Look for any full GC's.
 2.) Take some thread dumps during an incident.  Look for blocking.
 
 Dan
 
 
 
 below please find more information on versions and configs... let me know
 if more info is needed.
 
 OS Name:Linux (ubuntu 10.04.4)
 OS Version: 2.6.32-31-server
 Architecture:   amd64
 JVM Version:1.6.0_32-b05
 JVM Vendor: Sun Microsystems Inc.
 Server version: Apache Tomcat/6.0.24
 apache: Apache/2.2.14
 
 MPM configs
 START--
 IfModule mpm_prefork_module
  StartServers  5
  MinSpareServers   5
  MaxSpareServers  10
  MaxClients   26
  MaxRequestsPerChild  1000
 /IfModule
 END
 
 balancer conf
 START--
 Proxy balancer://mysite_balancer*
   Order deny,allow
   Allow from all
 /Proxy
 
 ProxyPassMatch ^/(.+.cf[cm])(.*)?$
 balancer://mysite_balancer/irised/client
 stickysession=JSESSIONID|jsessionid
 
 ProxyPassReverseCookiePath /irised /
 
 Proxy balancer://mysite_balancer
   BalancerMember ajp://localhost:8009 route=www1 retry=5
 /Proxy
 END
 
 connector xml
 START--
 Connector URIEncoding=UTF-8
  port=8009
  protocol=AJP/1.3
  connectionTimeout=2
  redirectPort=8443 /
 END
 
 
 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org
 
 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: ajp_ilink_receive error - please advise

2012-09-21 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Dan,

On 9/21/12 5:00 PM, Daniel Mikusa wrote:
 
 There are a few ways to [get a thread dump].

http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBdNDkACgkQ9CaO5/Lv0PC0WACgkd2+QU7MsT5diGB316NSP601
FZcAn26ek5nGMyhaKTnqr30g9plW/+oo
=y54N
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: ajp_ilink_receive error - please advise

2012-09-20 Thread Daniel Mikusa 
On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote:

 hello.
 
 we are running an application on tomcat and experiencing intermittent
 periods where the application is non-responsive and thus non-functional.
 the general set up is apache and tomcat, with mod_proxy_ajp as the
 connector (specific details and configs below).
 
 during these events, the primary apache error log shows the following:
 1. (104)Connection reset by peer: ajp_ilink_receive() can't receive header
 2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
 receive header
 
 it looks like error #1 repeats for a while and then error #2 repeats for a
 while.
 
 at the same time in the application specific apache error log (as specified
 in the vhost set up for app) we see the following errors repeated (though
 in mixed order from below):
 3. [error] ajp_read_header: ajp_ilink_receive failed
 4. [error] (120006)APR does not understand this error code: proxy: read
 response failed from (null) (localhost)
 5. [error] (120006)APR does not understand this error code: proxy: read
 response failed from 127.0.0.1:8009 (localhost)
 6. [error] [client X.X.X.X] proxy: error processing end
 
 and occasionally:
 7. [error] proxy: read zero bytes, expecting 464 bytes
 
 the catalina.out log registers nothing during the time period the
 application is unresponsive.
 
 a couple of other things to note:
 - these events are coming under light to no load as far as i can tell.
 - these events last from about 5 to 30 minutes and then everything works
 again as expected with no manual intervention.
 - the time of day of for the events is not consistent.
 - these events initially occured rarely, but over the last month have
 ramped up to daily.
 
 any suggestions on how to fix or further troubleshoot this problem? and
 thanks in advance for the help.

Here are a couple additional troubleshooting steps you can take on your Tomcat 
instances...

1.) Enable garbage collection logging.  Look for any full GC's.
2.) Take some thread dumps during an incident.  Look for blocking.

Dan


 
 below please find more information on versions and configs... let me know
 if more info is needed.
 
 OS Name:Linux (ubuntu 10.04.4)
 OS Version: 2.6.32-31-server
 Architecture:   amd64
 JVM Version:1.6.0_32-b05
 JVM Vendor: Sun Microsystems Inc.
 Server version: Apache Tomcat/6.0.24
 apache: Apache/2.2.14
 
 MPM configs
 START--
 IfModule mpm_prefork_module
   StartServers  5
   MinSpareServers   5
   MaxSpareServers  10
   MaxClients   26
   MaxRequestsPerChild  1000
 /IfModule
 END
 
 balancer conf
 START--
 Proxy balancer://mysite_balancer*
Order deny,allow
Allow from all
 /Proxy
 
 ProxyPassMatch ^/(.+.cf[cm])(.*)?$ balancer://mysite_balancer/irised/client
 stickysession=JSESSIONID|jsessionid
 
 ProxyPassReverseCookiePath /irised /
 
 Proxy balancer://mysite_balancer
BalancerMember ajp://localhost:8009 route=www1 retry=5
 /Proxy
 END
 
 connector xml
 START--
 Connector URIEncoding=UTF-8
   port=8009
   protocol=AJP/1.3
   connectionTimeout=2
   redirectPort=8443 /
 END


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: ajp_ilink_receive error - please advise

2012-09-20 Thread Django Radonich-Camp 
thank you for the replies chris and dan.

chris:
1.  upgrading apache is possible, but not easy (puppet manages the install).
2.  unfortunately i have not figured out how to reproduce the problem.  we
did extensive load testing on the system prior to launch and not once did
we see this problem...

dan:
1.  good idea.  do you know a good reference for setting this up?
2.  also, what is the process for capturing thread dumps?  this will be
difficult however because the incidents are random and so far, mostly
ocurring in the middle of the night.

thanks again for the help.


On Thu, Sep 20, 2012 at 5:35 AM, Daniel Mikusa dmik...@vmware.com wrote:

 On Sep 19, 2012, at 5:38 PM, Django Radonich-Camp wrote:

  hello.
 
  we are running an application on tomcat and experiencing intermittent
  periods where the application is non-responsive and thus non-functional.
  the general set up is apache and tomcat, with mod_proxy_ajp as the
  connector (specific details and configs below).
 
  during these events, the primary apache error log shows the following:
  1. (104)Connection reset by peer: ajp_ilink_receive() can't receive
 header
  2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
  receive header
 
  it looks like error #1 repeats for a while and then error #2 repeats for
 a
  while.
 
  at the same time in the application specific apache error log (as
 specified
  in the vhost set up for app) we see the following errors repeated (though
  in mixed order from below):
  3. [error] ajp_read_header: ajp_ilink_receive failed
  4. [error] (120006)APR does not understand this error code: proxy: read
  response failed from (null) (localhost)
  5. [error] (120006)APR does not understand this error code: proxy: read
  response failed from 127.0.0.1:8009 (localhost)
  6. [error] [client X.X.X.X] proxy: error processing end
 
  and occasionally:
  7. [error] proxy: read zero bytes, expecting 464 bytes
 
  the catalina.out log registers nothing during the time period the
  application is unresponsive.
 
  a couple of other things to note:
  - these events are coming under light to no load as far as i can tell.
  - these events last from about 5 to 30 minutes and then everything works
  again as expected with no manual intervention.
  - the time of day of for the events is not consistent.
  - these events initially occured rarely, but over the last month have
  ramped up to daily.
 
  any suggestions on how to fix or further troubleshoot this problem? and
  thanks in advance for the help.

 Here are a couple additional troubleshooting steps you can take on your
 Tomcat instances...

 1.) Enable garbage collection logging.  Look for any full GC's.
 2.) Take some thread dumps during an incident.  Look for blocking.

 Dan


 
  below please find more information on versions and configs... let me know
  if more info is needed.
 
  OS Name:Linux (ubuntu 10.04.4)
  OS Version: 2.6.32-31-server
  Architecture:   amd64
  JVM Version:1.6.0_32-b05
  JVM Vendor: Sun Microsystems Inc.
  Server version: Apache Tomcat/6.0.24
  apache: Apache/2.2.14
 
  MPM configs
  START--
  IfModule mpm_prefork_module
StartServers  5
MinSpareServers   5
MaxSpareServers  10
MaxClients   26
MaxRequestsPerChild  1000
  /IfModule
  END
 
  balancer conf
  START--
  Proxy balancer://mysite_balancer*
 Order deny,allow
 Allow from all
  /Proxy
 
  ProxyPassMatch ^/(.+.cf[cm])(.*)?$
 balancer://mysite_balancer/irised/client
  stickysession=JSESSIONID|jsessionid
 
  ProxyPassReverseCookiePath /irised /
 
  Proxy balancer://mysite_balancer
 BalancerMember ajp://localhost:8009 route=www1 retry=5
  /Proxy
  END
 
  connector xml
  START--
  Connector URIEncoding=UTF-8
port=8009
protocol=AJP/1.3
connectionTimeout=2
redirectPort=8443 /
  END


 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

ajp_ilink_receive error - please advise

2012-09-19 Thread Django Radonich-Camp 
hello.

we are running an application on tomcat and experiencing intermittent
periods where the application is non-responsive and thus non-functional.
 the general set up is apache and tomcat, with mod_proxy_ajp as the
connector (specific details and configs below).

during these events, the primary apache error log shows the following:
1. (104)Connection reset by peer: ajp_ilink_receive() can't receive header
2. (70007)The timeout specified has expired: ajp_ilink_receive() can't
receive header

it looks like error #1 repeats for a while and then error #2 repeats for a
while.

at the same time in the application specific apache error log (as specified
in the vhost set up for app) we see the following errors repeated (though
in mixed order from below):
3. [error] ajp_read_header: ajp_ilink_receive failed
4. [error] (120006)APR does not understand this error code: proxy: read
response failed from (null) (localhost)
5. [error] (120006)APR does not understand this error code: proxy: read
response failed from 127.0.0.1:8009 (localhost)
6. [error] [client X.X.X.X] proxy: error processing end

and occasionally:
7. [error] proxy: read zero bytes, expecting 464 bytes

the catalina.out log registers nothing during the time period the
application is unresponsive.

a couple of other things to note:
- these events are coming under light to no load as far as i can tell.
- these events last from about 5 to 30 minutes and then everything works
again as expected with no manual intervention.
- the time of day of for the events is not consistent.
- these events initially occured rarely, but over the last month have
ramped up to daily.

any suggestions on how to fix or further troubleshoot this problem? and
thanks in advance for the help.

below please find more information on versions and configs... let me know
if more info is needed.

OS Name:Linux (ubuntu 10.04.4)
OS Version: 2.6.32-31-server
Architecture:   amd64
JVM Version:1.6.0_32-b05
JVM Vendor: Sun Microsystems Inc.
Server version: Apache Tomcat/6.0.24
apache: Apache/2.2.14

MPM configs
START--
IfModule mpm_prefork_module
   StartServers  5
   MinSpareServers   5
   MaxSpareServers  10
   MaxClients   26
   MaxRequestsPerChild  1000
/IfModule
END

balancer conf
START--
Proxy balancer://mysite_balancer*
Order deny,allow
Allow from all
/Proxy

ProxyPassMatch ^/(.+.cf[cm])(.*)?$ balancer://mysite_balancer/irised/client
stickysession=JSESSIONID|jsessionid

ProxyPassReverseCookiePath /irised /

Proxy balancer://mysite_balancer
BalancerMember ajp://localhost:8009 route=www1 retry=5
/Proxy
END

connector xml
START--
Connector URIEncoding=UTF-8
   port=8009
   protocol=AJP/1.3
   connectionTimeout=2
   redirectPort=8443 /
END

Re: ajp_ilink_receive error - please advise

2012-09-19 Thread Christopher Schultz 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Django,

On 9/19/12 5:38 PM, Django Radonich-Camp wrote:
 apache: Apache/2.2.14

That's 9 revisions out of date. Any chance you can upgrade to 2.2.23
and re-test? Is it reliably reproducible?

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlBaTyUACgkQ9CaO5/Lv0PDGRQCdGr9fhj2dyiH4hxysDlyrGMH/
KN0An3tbQN6JTRPQfJnEt0VRvOYz9wSl
=WMXH
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org