apache-tomcat - APR does not understand this error code
We are on production with Apache/2.2.27 and Tomcat Version 7.0.55 on linux(2.6.32-431.37.1.el6.x86_64) with 6 core. what we're seeing is below error on webserver which has no pattern as to when it occurs. (104)Connection reset by peer: ajp_ilink_receive() can't receive header ajp_read_header: ajp_ilink_receive failed (120006)APR does not understand this error code: proxy: read response failed from : () it happens quite frequently when we do load test and on normal traffic it happens but rarely. As load ramps up we see cpu utilization going up(not hinting that this is reason) but we start seeing this error usually happens once we cross about 300 user load. we also have firewall in play between webserver and appserver. After we see above error webserver jumps from one app server to another app server. probably thinks original server is marked down. This has become very comman with load going up hence I took thread dump to see what's happening and I did not find any blocked threads. I can post whole thread dump if required here is unique thread stack trace where most of threads are under load. 1) "ajp-bio-8009-exec-3396" daemon prio=10 tid=0x7fa165b99800 nid=0x538b runnable [0x7fa069ad9000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:312) at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:367) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:118) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) - locked <0x0007d2998118> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 2) "pool-8-thread-1281" prio=10 tid=0x7fa0de140800 nid=0x5368 waiting on condition [0x7fa0721d9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00050e5f11f8> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 3) "PooledThread[50]" prio=10 tid=0x7fa0c4125000 nid=0x734 in Object.wait() [0x7fa0f064e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x0006915ccf60> (a de.hybris.platform.util.threadpool.PoolableThread) at java.lang.Object.wait(Object.java:503) at de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246) - locked <0x0006915ccf60> (a de.hybris.platform.util.threadpool.PoolableThread) at de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219) 4) "PooledThread[49]" prio=10 tid=0x7fa0b801f800 nid=0x733 in Object.wait() [0x7fa066cab000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x000653594388> (a de.hybris.platform.util.threadpool.PoolableThread) at java.lang.Object.wait(Object.java:503) at de.hybris.platform.util.threadpool.PoolableThread.resetAndReturnToPool(PoolableThread.java:246) - locked <0x000653594388> (a de.hybris.platform.util.threadpool.PoolableThread) at de.hybris.platform.util.threadpool.PoolableThread.run(PoolableThread.java:219) 5) "FD_SOCK client connection handler,hybris-broadcast,hybrisnode-2" daemon prio=10 tid=0x7fa0bc09d800 nid=0x67b7 runnable [0x7fa06b9f8000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.net.SocketInputStream.read(SocketInputStream.java:210) at org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1120) at java.lang.Thread.run(Thread.java:745) *Here's
Re: Tracking down memory leak
Please take a look at Memory Analyzer tool (http://www.eclipse.org/mat/). Run the app and take the heap dump while app is running and use the tool to analyze it. You could use VisualVM with plugins to get instrumentation or you could use hprof (http://docs.oracle.com/javase/7/docs/technotes/samples/hprof.html) HTH On 10/20/15 6:20 AM, David kerber wrote: I'm trying to track down the source of a memory leak in one of my applications. I have examined the code but have been unable to fix it, so am looking for some way of instrumenting my app while running on the server. What is the easiest/best (I realize those two criteria may not give the same answer!) way? Running TC 8.0.20-something, JRE 8.0.something recent, on windows Server 2012 R2. Thanks for any suggestions! Dave - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tracking down memory leak
I'm trying to track down the source of a memory leak in one of my applications. I have examined the code but have been unable to fix it, so am looking for some way of instrumenting my app while running on the server. What is the easiest/best (I realize those two criteria may not give the same answer!) way? Running TC 8.0.20-something, JRE 8.0.something recent, on windows Server 2012 R2. Thanks for any suggestions! Dave - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
