[
https://issues.apache.org/jira/browse/AMQ-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15632620#comment-15632620
]
Christopher L. Shannon commented on AMQ-6482:
---------------------------------------------
>From what I can tell, the issue seems to be that the Websocket connections are
>hanging waiting for a response from the client when doing the inactivity
>check. There are tons of the below blocked threads:
{noformat}
"ActiveMQ InactivityMonitor Worker" #40006 daemon prio=5 os_prio=0
tid=0x00007fc60cfd4800 nid=0x4c92 waiting on condition [0x00007fc4148c7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000055a702d70> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219)
at
org.eclipse.jetty.websocket.common.BlockingWriteCallback$WriteBlocker.block(BlockingWriteCallback.java:83)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:107)
at
org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:387)
at
org.apache.activemq.transport.ws.jetty9.StompSocket.sendToStomp(StompSocket.java:47)
at
org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:888)
at
org.apache.activemq.transport.ws.AbstractStompSocket.oneway(AbstractStompSocket.java:63)
at
org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:334)
at
org.apache.activemq.transport.AbstractInactivityMonitor.access$800(AbstractInactivityMonitor.java:41)
at
org.apache.activemq.transport.AbstractInactivityMonitor$4.run(AbstractInactivityMonitor.java:200)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
There seems to me to be 2 issues here:
1) Jetty seems to have an issue where it gets stuck and never returns or times
out the blocking send. I found this issue here:
https://github.com/eclipse/jetty.project/issues/272
2) The inactivity monitor is supposed to be doing a one way send anyways but
the send is actually a blocking send so it can get stuck waiting for a
response.
I think under the old Jetty 8 websocket stuff we were doing async sends but
under Jetty 9 that got switched to sync and it waits for the bytes to transfer.
I think, at least for the inactivity monitor and maybe all calls, we should be
doing async sends and not blocking so that the lock can be released. Or at
least have some way to timeout.
[~tabish121], what do you think about something like this?
https://github.com/cshannon/activemq/commit/915cc828b928268e18bb50b99247e78e77aa0d51
> Possible thread leak ?
> ----------------------
>
> Key: AMQ-6482
> URL: https://issues.apache.org/jira/browse/AMQ-6482
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.14.1
> Reporter: Maxence Dunnewind
> Labels: threading
> Attachments: thread_dump
>
>
> We have an activemq broker running for ~ 24 hours, handling ~ 100k topics and
> up to 8k connections (for ~ 24k consumers).
> Looking at my graph, I see that I get a big increase of threads count
> happended (in 4 steps, ~ 500 more threads each time) as shown here :
> https://snapshot.raintank.io/dashboard/snapshot/ExUb4pgNYnmBo92JgbZrvJ6fiBm3PNrI
> However, even if the traffic dropped in the night, the thread count never
> decreased and is still ~ 2500.
> I checked a thread dump and it looks like (full dump attached) :
> ```
> # grep Thread.State thread_dump |awk '{ print $2 }' |sort|uniq -c
> 26 RUNNABLE
> 70 TIMED_WAITING
> 2446 WAITING
> ```
> Mhh, so many WAITING threads, let's check why :
> ```
> # grep 'parking to' thread_dump |cut -d '>' -f 2 |sort |uniq -c
> 1 (a java.util.concurrent.CountDownLatch$Sync)
> 1308 (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 1159 (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 41 (a java.util.concurrent.SynchronousQueue$TransferStack)
> ```
> I'm not a big java expert but seems a bit unexpected behavior to me.
> For various reason (big number of topics etc ) we use custom GC config, here
> it is :
> ```
> ACTIVEMQ_OPTS_MEMORY="-Xms128M -Xmx10G -Xloggc:/home/log/activemq/gc.log
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=75 -XX:NewRatio=4"
> ```
> Maxence
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
