Update:
I was using Stomp protocol for both the Publishers and Consumers. A
check with jconsole showed me that there were around 20,000 threads. To
test it further, I modified my stomp library (written in perl), to close
connection and re-connect, every 100 messages. Activemq, consistently
stopped responding, and the memory used was around 400MB (with another
500 MB of physical memory free) and the number of threads were 20,000.
On selecting the MBean tab in jconsole, I got java.net.connection exception.
For the next test, I used the ProducerTool (bundled with the activemq
binaries). The only change was that the ProducerTool would run in a
while(true){} loop. The idea was to see if/when activemq would break.
Again, there were 8 Producers and 3 Consuemers (of which, one of them
was the bundled ConsumerTool and the remaining used Stomp). Around
400,000th message (roughly the same number on all 8 processes), the
jconsole showed non-heap memory used as around 400MB and the number of
threads at 200, when activemq stopped responding. I had enabled DEBUG
messages, but no error message were reported in the log. I was able to
use jconsole to look into the enqueue and dequeue count.
This scenario can be duplicated using the bundled ProducerTool and
ConsumerTool. I guess the average message/second was 150 per Producer
process. Also a live environment can easily have 8 or more producers
(although not with such a high message/second rate), also this set of
tests show me that the Stomp implementation is not as stable as desired.
Does anyone here use activemq on a production server, if so under what
load and what is the availability of the activemq service ? How can I
setup a stable activemq server in a high availability manner. Thanks
- Sandeep
Sandeep Chayapathi wrote:
Hi all,
Recently I noticed that the activemq would stop sending message to
the consumers after some time. Here is a unscientific test to
duplicate this issue:
The test involved creating around 7 durable subscriptions and
ctrl+c'ing them. Once the durable subscriptions were registered,
around 8 publisher processes were started, each process wrote a
'fortune cookie' , in an infiniteloop, to the activemq server. In
parallel, 3 subscriber processes were started. The 3 subscribers were
randomly stopped and started, cyclically. At about 15000 messages (on
each of the 8 write processes), activemq stopped sending the message
to to the subscribers and the publisher processes hung on "syswrite"
to activemq. Then another publisher process was started, activemq
actually accepted messages, but none of the subscribers received it.
The next test involved, using mysql as the data store (instead of the
default journalled db). The same steps were followed and at about
18000 messages (on each of the 8 processes), activemq stopped sending
messages to the subscribers.
Im using activemq 4.0.2. Any suggestions/help is appreciated. Thanks
- Sandeep
