We are on production with Apache/2.2.27 and Tomcat Version 7.0.55 on
linux(2.6.32-431.37.1.el6.x86_64) with 6 core. what we're seeing is below
error on webserver which has no pattern as to when it occurs.
(104)Connection reset by peer: ajp_ilink_receive() can't receive header
ajp_read_header: ajp_ilink_receive failed
(120006)APR does not understand this error code: proxy: read response
failed from <server ip>:<port> (<server ip>)
it happens quite frequently when we do load test and on normal traffic it
happens but rarely. As load ramps up we see cpu utilization going up(not
hinting that this is reason) but we start seeing this error usually happens
once we cross about 300 user load.
we also have firewall in play between webserver and appserver.
After we see above error webserver jumps from one app server to another app
server. probably thinks original server is marked down. This has become
very comman with load going up hence I took thread dump to see what's
happening and I did not find any blocked threads. I can post whole thread
dump if required here is unique thread stack trace where most of threads
are under load.
1) "ajp-bio-8009-exec-3396" daemon prio=10 tid=0x00007fa165b99800
nid=0x538b runnable [0x00007fa069ad9000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:312)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:367)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:118)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
- locked <0x00000007d2998118> (a org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
2) "pool-8-thread-1281" prio=10 tid=0x00007fa0de140800 nid=0x5368
waiting on condition [0x00007fa0721d9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000050e5f11f8> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3) "PooledThread[50]" prio=10 tid=0x00007fa0c4125000 nid=0x734 in
Object.wait() [0x00007fa0f064e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006915ccf60> (a
de.hybris.platform.util.threadpool.PoolableThread)
at java.lang.Object.wait(Object.java:503)
at
de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246)
- locked <0x00000006915ccf60> (a
de.hybris.platform.util.threadpool.PoolableThread)
at
de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219)
4) "PooledThread[49]" prio=10 tid=0x00007fa0b801f800 nid=0x733 in
Object.wait() [0x00007fa066cab000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000653594388> (a
de.hybris.platform.util.threadpool.PoolableThread)
at java.lang.Object.wait(Object.java:503)
at
de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246)
- locked <0x0000000653594388> (a
de.hybris.platform.util.threadpool.PoolableThread)
at
de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219)
5) "FD_SOCK client connection handler,hybris-broadcast,hybrisnode-2"
daemon prio=10 tid=0x00007fa0bc09d800 nid=0x67b7 runnable
[0x00007fa06b9f8000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at
org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1120)
at java.lang.Thread.run(Thread.java:745)
*Here's our AJP connector config on tomcat*
<Connector protocol="AJP/1.3"
port="8009"
redirectPort="443"
useIPVHosts="true"
URIEncoding="UTF-8"
maxThreads="850"
minSpareThreads="100"
connectionTimeout="30000"
backlog="1000"
keepAliveTimeout="60000"
maxConnections="700"
acceptorThreadCount="4"/>
*And below is apache conf*
prefork.c
StartServers 20
MaxSpareServers 40
MinSpareServers 40
ServerLimit 700
MaxClients 700
IfModule
*and lb.conf*
BalancerMember ajp://<host_ip>:8009 route=app5 retry=60
ProxySet stickysession=JSESSIONID
ProxySet lbmethod=byrequests failonstatus=503,504
ProxySet disablereuse=On
Any help is appreciated.
