[
https://issues.apache.org/jira/browse/AMQ-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630685#comment-15630685
]
Maxence Dunnewind commented on AMQ-6482:
----------------------------------------
New websocket connections are definitively created quiet often. They are
created every time a user log in to the website (or throug mobile app) or
switch to a new page while being logged in.
Something (maybe) interesting is that we had a bug in our code which was
creating too many topics (~ 100k in a few seconds) by sending messages to lot
of these new topics. 95% of them were never read. We fixed it and reduced to ~
8000 topics (no GC of old destination right now), but the thread leak is still
present and seems to grow at same speed than before (from ~ 300 right after
start to ~ 1000 in 3 hours), so looks like this is not related to
topics/messages. So the only remaining possibility is client side. Moreover,
looking at the graph, it seems that the number of thread increases by "step".
For ex between 7.50 and 8.10 pm (utc+2) it only increase of 100threads, then
in 15 minutes I have a 500 thread bump, then almost stable for 10 minutes, then
+400 threads in 5 minutes etc ... It seems that the number of threads increases
faster when I have a lot of "outgoing messages", and slowly when I don't have
much, maybe a hint.
Though this is a production service, we can accept small downtime (like for a
restart, if config update is needed) and I'm available as Sp4rKy on freenode if
you want some live debug (baiscally between 9 and 19 utc+2).
> Possible thread leak ?
> ----------------------
>
> Key: AMQ-6482
> URL: https://issues.apache.org/jira/browse/AMQ-6482
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.14.1
> Reporter: Maxence Dunnewind
> Labels: threading
> Attachments: thread_dump
>
>
> We have an activemq broker running for ~ 24 hours, handling ~ 100k topics and
> up to 8k connections (for ~ 24k consumers).
> Looking at my graph, I see that I get a big increase of threads count
> happended (in 4 steps, ~ 500 more threads each time) as shown here :
> https://snapshot.raintank.io/dashboard/snapshot/ExUb4pgNYnmBo92JgbZrvJ6fiBm3PNrI
> However, even if the traffic dropped in the night, the thread count never
> decreased and is still ~ 2500.
> I checked a thread dump and it looks like (full dump attached) :
> ```
> # grep Thread.State thread_dump |awk '{ print $2 }' |sort|uniq -c
> 26 RUNNABLE
> 70 TIMED_WAITING
> 2446 WAITING
> ```
> Mhh, so many WAITING threads, let's check why :
> ```
> # grep 'parking to' thread_dump |cut -d '>' -f 2 |sort |uniq -c
> 1 (a java.util.concurrent.CountDownLatch$Sync)
> 1308 (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 1159 (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 41 (a java.util.concurrent.SynchronousQueue$TransferStack)
> ```
> I'm not a big java expert but seems a bit unexpected behavior to me.
> For various reason (big number of topics etc ) we use custom GC config, here
> it is :
> ```
> ACTIVEMQ_OPTS_MEMORY="-Xms128M -Xmx10G -Xloggc:/home/log/activemq/gc.log
> -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=75 -XX:NewRatio=4"
> ```
> Maxence
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
