I repeat my earlier suggestion that you should use a sampler on all brokers to characterize where the time is being spent.
The fact that turning logging to debug causes the broker to experience a slowdown almost immediately makes it seem like maybe there's an issue with disk I/O or space, but that's just a guess and using a sampler will give you something more concrete than a guess. Also, you've characterized the problem as messages getting "stuck" in the network connectors. When this happens, are individual messages truly stuck (i.e. no messages are being passed), or is it simply that the rate they're flowing out at os lower than the rate they're flowing in at (so there's a net backup but individual messages are still being passed)? And when messages are passed, do they arrive in order, or do they show up vastly different from the order in which they were sent? Tim On Jul 11, 2017 4:29 PM, "sravan" <[email protected]> wrote: > Unfortunately we are in a predicament where we have issues in reproducing > the > problem in performance test environments as well as debugging. ActiveMQ > INFO > logs does not contain much information to understand what's going on. When > we turn on DEBUG logging (in lower env) ActiveMQ is hanging just in few > minutes and never comes back. So far the only one clue we are relying on > is, > while monitoring ActiveMQ consoles in our prod environment, we observed > messages stuck on network bridge for multiple hours. Also in our research, > we know that consumers are doing fine and the main issue is with ActiveMQ > which is delivering messages very late and we simply don't know the root > cause. When we worked with an ActiveMQ consultant a year ago, we were > advised to scale ActiveMQs vertically rather than scaling them > horizontally. i.e. Have 2 larger AMQ nodes rather than having 4 smaller > nodes in a cluster. We were told that this avoids many potential anomalies > with network brdige and shipping messages across the nodes etc...Does > anyone > here have any suggestions with respect to how else we could debug or fix > our > issue. Again just to refresh your minds, the main issue we’ve been > struggling with is...Our AMQs are delivering messages extremely late, > sometimes a day or even two days later. We do not see any JMS exceptions, > and no exceptions or issues at consumer's end. Any inputs are greatly > appreciated. > > > > -- > View this message in context: http://activemq.2283324.n4. > nabble.com/Messages-are-stuck-in-ActiveMQ-5-11-and- > delivered-for-after-more-than-24-hours-tp4727694p4728468.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
