So this a little TL;DR... but I try answer some of your questions inline...
On Mon, Mar 31, 2014 at 11:53 AM, Tom_Z <tomatgooglec...@arcor.de> wrote: > Hi there, > > as an ActiveMQ starter I would like you to give me some feedback on my > setup. > > Scenario is: > - currently single broker setup with standalone ActiveMQ 5.8 using LevelDB, > JRE 7, Protocol for all connected systems will be OpenWire > - 2 backend systems, let's call them B1 and B2, acting as producer and > consumer in sequential, non-multithreaded working mode > - B1 and B2 do produce different kinds of messages into their own individual > queue (two 'in'-queues from the client perspective) > - these systems can rely on a stable network connection and can be seen as a > fast producer and consumer running all over the day > - on the other side we do have a client application C with up to 250 > instances working in parallel as a consumer on the two before mentioned > out-queues > - each client C does produce different kinds of response messages to > different 'out'-queues > - the clients do not have a stable network connection and messages can be > stuck in the queue for a while resulting in large out queues, at least quite > a bundle of messages for some of the consumers can remain for a longer time > - expected data volume: 50.000 messages a day, average size 10kb (range is > from 1kb to 1MB) > - all messages must be persisted > - all queues and virtual destinations are predefined in activemq.xml > > Message processing: > - B1 produces into QIB1 (in-queue for B1) and consumes from QO1, QO2, QO3 > (queues-out 1 to 3), spread all over the day > - B2 usually produces large amounts of messages only once a day into QIB2, > but consumes continiously returning messages from QO4 > - every client C consumes request messages designated to that individual > client through JMSXGroupID from QIB1 and QIB2 and produces response messages > to QO1,2,3,4 > - in the opposite direction B1 and B2 are able to process messages for > theirselves delivered to their designated queues > - message order is relevant in both directions (in and out) > - for testing purposes und to be able to redeliver messages to clients in > case of loss at client side we have defined "Composite Destinations", so > that each message is copied (forwardOnly="false") into an in- respectively > out-archive queue, so that messages can be pushed back into the relevant > queues (we are a little concerned about the quality of implementation on the > client side and finding bugs processing messages in production and like to > be prepared for such cases, archiving will be deactivated later) > > 1. Is that a suitable setup to use ActiveMQ in production? If I read correctly, you have only one instance running? No.. this is not a suitable setup for production. What happens if/when that one instance fails? You should think about having at least a Slave broker in warm standy. > > 2. Is a composite destination from queue to queue okay? In the documentation > often topics are used, from the archive queues will never be consumed - only > copied, deleted or purged and viewed in the queue browser. The messages will never be consumed? This will not work for long periods, you will run out of disk space. Why not just store to a separate, longer-term storage? Sounds like you're trying to use ActiveMQ for both a real-time even broker and some kind of longer-term audit log... not a good idea. > > 3. Is it possible to add an expiry date while forwarding to the archive > queue/topic (Apache Camel is not in use)? > > 4. One open question is whether to use a single out queue (easier for client > app C) and to use filtered destinations, filtering must then base on a > application specific message property to distribute the message types to the > different out-queues, is that possible? Does it perhaps perform better, on > client and server side? Depends on how many messages and how much you're filtering...If the producers know the partition ahead of time, then just stick with that until it doesn't scale/work. > > 5. I still have to take a deeper look at the memory settings. The queues do > have quite different characteristics, from being constantly emptied (out > queues) to holding large amounts of messages for lots of clients for some > time being able to deliver at any time (in queues). I guess at least > regarding to both out queues containing messages for up to 250 different > message groups I have to do some finetuning? > > 6. What are the side effects of too little memory for queue? A couple things... message groups are honored based on messages that have been paged into memory. http://activemq.apache.org/message-groups.html If you have too little memory allocated to broker or destinations, then messages will need to be paged in from disk (which can be slower). Out-queues are > not time or performance critical and B1 and B2 should grab all messages > regardless of the message group, in-queues must deliver fast to the client > according to the requested message group. > > I found a note: "like message selector matching, grouping based on > JMSXGroupID occurs before dispatch on messages in memory. With the default > maxPageSize setting, large backlogs of messages destined for one group they > can block receipt of messages to other groups if they don't all fit in > memory." > > That means if memory is not enough for all(!) groups/messages in the > in-queue, a client possibly receives only a subset of messages for his > message group? > > Thanks for any hints and advice, > Tom > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Questions-to-broker-architecture-and-setup-memory-configuration-tp4679749.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. -- Christian Posta http://www.christianposta.com/blog twitter: @christianposta
