Hi, I am trying to find the best scalability setup regarding my JMS scenario, and I would like to ask you to help me to do that. The scenario is very simple – I have a set of producers, which send thousands of persistent messages on a queue for a several seconds, as a consequence of some UI activity. I also have a set of consumers, responsible for message processing. The order of message processing is not important – each message is self contained, and its processing is completely independent of the other’s messages processing. The processing is a very complex, and typically it takes one or several minutes to complete the message processing.
In order to improve my application scalability, I would like to use several independent brokers and delegate the decision to the sender application which broker would to send a message to. I don’t think to use a single broker, /despite of the fact that this is a highly recommended solution/ because I don’t want to have a single point of failure. Network of brokers is a useful option for me, but I am afraid that this will introduce addition network overhead. This “sender side manual load balancing” combines the best of the both worlds – no single point of failure and no cluster communication overhead. Probably this scenario is very similar to the jedi-transport request: https://issues.apache.org/activemq/browse/AMQ-816 However, jedi-transport is still unavailable, so it is not a real option. In order to delegate the decision to choose a broker to which to send a message, the sender application should be able to know the list of all available brokers. One possible solution is to use a hard coded list, however a more advanced solution will be able to detect dynamically available brokers. The idea is to implement discovery listener on the client side and to listen for discovery events and to maintain a set of available brokers. The sender application will be able to send a message using round robin load balancing between brokers. On the other hand I need to use broker’s discovery mechanism, but with some modification – I expect brokers to expose themselves, but not to establish a network of brokers between them. I checked the code, and regarding my findings the only possible way to do that is to setup each broker in different group and to share one and the same multicast address. For example: Broker1 -> discoveryUri="multicast://default?group=broker1" Broker2 -> discoveryUri="multicast://default?group=broker2" … BrokerN -> discoveryUri="multicast://default?group=brokerN" This setup will challenge all brokers to advertise themselves and at the same time each broker belongs to its own group. I don’t expect additional network traffic due to clustering, except the packets sent by discovery agents. This will provide a solution for membership detection to the sender application – it simply listens on the same multicast address and analyses the incoming notifications. For example: received: broker1.ActiveMQ-4.alive.%localhost%tcp://mymachine:61616 received: broker2.ActiveMQ-4.alive.%localhost%tcp://mymachine:61618 received: broker1.ActiveMQ-4.alive.%localhost%tcp://mymachine:61616 received: broker1.ActiveMQ-4.dead.%localhost%tcp://mymachine:61616 received: broker2.ActiveMQ-4.alive.%localhost%tcp://mymachine:61618 received: broker2.ActiveMQ-4.dead.%localhost%tcp://mymachine:61618 Probably this is very tricky and risky solution, since the client side application should analyze a data packets, used only internally. I need to re-implement the logic of MulticastDiscoveryListener with the main aim to implement DiscoveryListener. The implementation of this listener will receive notification about service start/stop and will maintain a set of the available subscribers. The sender application will be able to discover dynamically all available brokers and to use some load balancing strategy to utilize them. Do you think that my approach is siutable? Do you see any serious downfalls in my suggestion? Any help will be highly appreciated. Greetings -- View this message in context: http://old.nabble.com/sender-side-manual-load-balancing-tp28903218p28903218.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
