I use an application called Spacewalk to manage RHEL systems in an
enterprise environment. The application provides an API that I use for
automation purposes. While load starts to increase on the application
server, we reach a breaking point where the application becomes
unresponsive, and throws 500 internal server errors. Listed at the bottom
are some of the errors I see when this happens. I'm looking for advice on
how to better diagnose and\or tune my settings to optimize Tomcat
performance. I feel like the system is beefy enough to handle this load,
but Tomcat appears to be my bottleneck.. How should I go about resolving
this? I have played around with the AJP connector settings quite a bit, but
can't seem to find suitable parameters.
OS: RHEL6.6
RAM: 64GB (Please note, under heavy load, we are not utilizing
even 50% of RAM.. it seems to be all CPU, but I'm not sure how to get
Tomcat to utilize more RAM other than boosting the Xmx settings which I
have already done)
CPU: 16 (vCPU)
Tomcat:
apache-tomcat-apis-0.1-1.el6.noarch
tomcat6-el-2.1-api-6.0.24-83.el6_6.x86_64
tomcat6-lib-6.0.24-83.el6_6.x86_64
tomcat6-servlet-2.5-api-6.0.24-83.el6_6.x86_64
tomcat6-6.0.24-83.el6_6.x86_64
tomcat5-jsp-2.0-api-5.5.27-7.jpp5.noarch
tomcat6-jsp-2.1-api-6.0.24-83.el6_6.x86_64
tomcat5-servlet-2.4-api-5.5.27-7.jpp5.noarch
Java:
java-1.6.0-openjdk-1.6.0.36-1.13.8.1.el6_7.x86_64
Oracle client:
oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64
JAVA_OPTS=-ea -Xms512m -Xmx4096m -Djava.awt.headless=true
-Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser
-Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024 -XX:MaxNewSize=256
-XX:-UseConcMarkSweepGC -Dnet.sf.ehcache.skipUpdateCheck=true
-Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory
-Djava.library.path=\${system_property:java.library.path}:/usr/lib:/usr/lib64/oracle/11.2/client/lib
Connector settings:
Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2
redirectPort=8443 URIEncoding=UTF-8 address=127.0.0.1
maxThreads=1024 maxKeepAliveRequests=1000/
!-- Define an AJP 1.3 Connector on port 8009 --
Connector port=8009 protocol=AJP/1.3 redirectPort=8443
URIEncoding=UTF-8 address=127.0.0.1 maxThreads=1024/
Connector port=8009 protocol=AJP/1.3 redirectPort=8443
URIEncoding=UTF-8 address=::1 maxThreads=1024/
From /var/log/tomcat6/catalina.out
Aug 25, 2015 1:33:23 AM org.apache.jk.core.MsgContext action
WARNING: Unable to send headers
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:532)
at
org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
at org.apache.jk.core.MsgContext.action(MsgContext.java:266)
at org.apache.coyote.Response.action(Response.java:183)
at org.apache.coyote.Response.sendHeaders(Response.java:379)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288)
at
org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:95)
at
org.apache.jasper.runtime.JspWriterImpl.flush(JspWriterImpl.java:175)
at
org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:956)
at
org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:622)
at
com.opensymphony.module.sitemesh.taglib.page.ApplyDecoratorTag.doEndTag(ApplyDecoratorTag.java:258)
at
org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspx_meth_page_005fapplyDecorator_005f0(Unknown
Source)
at
org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspService(Unknown
Source)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:438)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:415)
at
org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:342)
at
org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:286)
at
