I mean 50,000-100,000 messages are there in queue after which it crashes and this range is not definitive, it some crashes below a pile of 50,000 messages too. Actually, I am not able to predict when exactly it is crashing, if its is crashing due to CPU usage or memory usage or IO.
Logs are behaving fine before the crash, so I believe everything is fine with permissions and log level. Log rotation would not remove the activemq.log completely. Further, after crash, activemq process is still alive (checked using ps command), however, Admin UI (port 8161) is not available after crash. On Mon, Nov 3, 2014 at 7:03 PM, James Green <james.mk.gr...@gmail.com> wrote: > On 3 November 2014 09:42, Vikas Agarwal <vi...@infoobjects.com> wrote: > > > Hi, > > > > We are using ActiveMQ for 3+ years, however, we didn't test it under > heavy > > load. Recently, we started using ActiveMQ in another project where > > sometimes load increases exponentially as we are listening to twitter > > stream which can pile up a lot of messages depending of occurrence of > some > > global event like iPhone 6 launch. Now, what we are seeing in ActiveMQ > > admin UI is that one of the queues gets piled up with thousands of > message > > ranging from 50k-100k. Now, sometimes (almost daily) the ActiveMQ crashes > > > > Do you mean: > > 1. The queues have 50-100,000 messages each, or > 2. The queues have thousands of messages each of which may be 50-100k in > size? > > > > in between of my sleeping hours. :( And the strange thing is that when > try > > to view the log file for the reason of the crash, I am not able to find > the > > activemq.log file at all. We earlier faced ActiveMQ crash issues, but > that > > > > Ensure you have INFO logging switched on and that AMQ writes to this log on > start-up. Also, that log could be rotated - ensure the process has > permission to write to it. > > > > was due to producer flow control and we disabled that to fix the issue > and > > main point is that in those cases we never had the case where the log > file > > was missing. I have to restart the process to fix the issue and that too, > > twice because on first restart, it shows error about missing or corrupt > > data log. > > > > So, please suggest me where to look for such issue or what should be my > > next move for debugging. I am stuck with missing log file. > > > > For information here are the memory settings: > > > > <systemUsage> > > <systemUsage> > > <memoryUsage> > > <memoryUsage limit="*256 mb*"/> > > </memoryUsage> > > <storeUsage> > > <storeUsage limit="*100 gb*"/> > > </storeUsage> > > <tempUsage> > > <tempUsage limit="*50 gb*"/> > > </tempUsage> > > </systemUsage> > > </systemUsage> > > > > You might want to start here and work out the model of usage you're seeing: > http://activemq.apache.org/javalangoutofmemory.html > > For instance: thousands of messages, few clients; or thousands of messages > and lots of clients. > -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
