If you're able to analyze the snapshot, that's great. If you want me to, I can. If you're using any custom broker plugins, it's possible that your package, class, and method names will appear in the stack trace information captured by the snapshot, so if you're using custom plugins, make sure you consider whether any of that is sensitive before sharing it publicly on this list.
Ideally you'd only capture the snapshot while the problem is occurring; the more time you capture from when everything is normal, the harder it will be to zero in on the behavior from when things are bad. So I recommend capturing for a while, then if the problem hasn't started occurring after an hour or two, stop the sampling session and start a new one. Tim On Feb 2, 2018 7:57 AM, "Stefanic" <snico...@movingintelligence.com> wrote: > I will try to remember doing so, are you interested in the CPU sampling > snapshot? And if so over what amount of time? > > (I'm not very experienced with profiling/sampling, only really done > heapdump > and threaddump analysis) > > > Tim Bain wrote > > Would you be able to use JVisualVM to perform CPU sampling while this is > > occurring, to find out what those threads are doing? **WARNING** Do NOT > do > > CPU *profiling*, which would slow your broker significantly; sampling is > > lightweight and generally held to be safe to do against an operational > > process, but profiling is very heavyweight. Make sure you're on the right > > tab in JVisualVM. > > > > Tim > > > > On Feb 2, 2018 12:05 AM, "Stefanic" < > > > snicodem@ > > > > wrote: > > > >> Hi Tim, > >> > >> There are no selectors, so there are no messages left behind. > >> We have seen this three times now in production and every time when the > >> queue is empty (after about 30 minutes of refreshing) the problem goes > >> away. > >> > >> Threads behavior is rather strange, when the blocking issue is on-going > >> we > >> see 1 thread constantly running (100%) in visualvm, and all other > threads > >> (31 in our case) almost seem synchronized on the same object because all > >> of > >> them start running at the same time and go into timed_waiting state at > >> the > >> same time. > >> That results in 33% running of all other threads and that is not enough > >> to > >> keep our queue empty. > >> > >> Reproducing this behavior is nearly impossible for us, it occurs > randomly > >> within about a week. > >> > >> Here are the things we tried so far: > >> - Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1 > >> but > >> the problem occurred again on that version so we assume it is not the > >> broker/server > >> - Yesterday we downgraded both ActiveMQ and Camel client libraries: > >> ActiveMQ > >> down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather > >> only downgraded 1 at a time but it's our production environment so > cannot > >> play around too much) > >> > >> I will reply here if we experience the blocked state again. > >> > >> > >> > >> -- > >> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > >> f2341805.html > >> > > > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User- > f2341805.html >