The functionality you're contemplating for the broker plugin seems pretty far off from the standard ActiveMQ approach, since it involves knowing the state of a message across the full network of brokers rather than allowing each broker to operate as a stand-alone island that only needs to know about itself and how to pass message to its neighbors. So I'd be very hesitant to go down the path you're suggesting.
And I suspect you won't have to; the functionality you're looking to use is how ActiveMQ should work, so I think you'll find that either there's a bug in ActiveMQ or a bug in your configuration. Or maybe there's a minor missing feature or one that's not been applied to all the transports it should have; either way, I suspect the actual fix will turn out to be much simpler than the plugin you're contemplating. We'll see what shakes out as you simplify your configuration to remove the less-common configuration options. Eventually we should get to a configuration that delivers the messages correctly but maybe doesn't have all the configuration options enabled that you want to use, and then it's a question of why it stops working with the less-common configuration options you're using. On Tue, Nov 4, 2014 at 8:16 AM, Andreas Gies <andr...@wayofquality.de> wrote: > Hi Tim, > > I will make some time tomorrow to try your suggestions. > > A Virtual destination would simply replace a durable subscriber with a > Queue receiver. > Messages posted to a mapped topic would end up in the the underlying > queues. > > Now, a consumer that wasn't connected to the peer broker would immediately > receive > the messages on the broker - that would address the problem of temporarily > unvisible > messages. > > As for the duplicated delivery of messages - effectively we would have > concurrent consumers > (even though most likely onlz one of the would be connected at any one > point in time). > A message consumed on either side of the NWOB would effectively vanish > from the underlying > Queue and not be delivered again. > > > Due to that I was contemplating a distributed broker plugin that would > kind of synchronize the > consumption of messages of a DS within a NWOB. > > However, before going that path I wanted to make sure that I am not > missing something obvious. > > > Thanks for the answers so far and best regards > Andreas > > On 03/11/14 20:23, Tim Bain wrote: > >> Andreas, >> >> Your spec added two configuration elements your previous post didn't >> mention, and I'd like to eliminate each of them in turn to see if it's >> causing/contributing to the problem. >> >> 1. Your networkConnectors are apparently multicast. Please see what >> happens if you configure them as >> static:(tcp://host:port?tcpOptions)?staticOptions, to take the >> multicast >> (and the broker discovery that it's presumably doing) out of the >> equation. >> I recently experimented with what happens when the failover is >> allowed to >> perform a reconnect in a broker-to-broker networkConnector, and the >> result >> is duplicate and/or stale subscriptions between the brokers. That >> behavior >> could explain what you're seeing, if multicast is similarly performing >> reconnects without notifying the static wrapper so it can recreate the >> network bridge, so let's take it out of the equation to see if the >> behavior >> changes. (I've never used multicast, so this might not make sense; if >> someone knows that this can't be the issue, please say so.) >> 2. I don't know how gracefully conduitSubscriptions reacts to >> consumers >> >> moving around the network of brokers; I don't believe this should be >> the >> problem, but if #1 doesn't produce any change in behavior, can you set >> conduitSubscriptions=false and see if anything changes? >> >> I'm not clear on how Virtual Topics will solve the problem; can you >> explain? To me this feels like a problem with broker-to-broker management >> of subscriptions made on behalf of clients (most likely duplicate >> subscriptions for a client, one from each broker, after a failover), and >> I'm not sure how a Virtual Topic would make it any better if that's the >> case. But if you know of a way that it would, that might help me to >> understand what's going on. >> >> Tim >> >> On Sat, Nov 1, 2014 at 9:50 AM, Andreas Gies <andr...@wayofquality.de> >> wrote: >> >> Hello Tim, >>> >>> thanks for your answer. It took me a bit to digest it - so my apologies >>> for >>> the delay in my answer. >>> >>> I have come up with a test case that shows - and unfortunately confirms >>> my observations.The test case is located at [1]. >>> >>> Here is the excerpt of my problem descriptions & observations: >>> >>> /** >>> * This specification shall help to investigate the duplicate delivery >>> of >>> messages for durable subscribers >>> * within a network of brokers. The problem has been posted on the >>> ActiveMQ mailing list on Oct. 18th 2014 >>> * and was described as follows: >>> * >>> * Suppose you have a network of brokers consisting of two members >>> discovering each other via multicast. >>> * The network bridge is set up using conduit subscriptions. Now assume >>> that we have a durable subscriber >>> * named "S" that connects to the network of brokers using a failover >>> uri >>> pointing to both brokers. >>> * >>> * First, the subscriber connects to Broker A. It will consume all >>> messages published to either Broker A or B. >>> * Now the subscriber disconnects and stays offline for a bit, then it >>> reconnects to Broker B. Now it will pick >>> * up all messages that have been published while it was offline. >>> * >>> * Let's say then 10 messages are published. All is well as the >>> subscriber >>> consumes those messages. >>> * If the subscriber then disconnects and reconnects to Broker A, these >>> 10 >>> messages will be consumed >>> * again by the reconnected subscriber. >>> * >>> * According to Tim Bain on Oct., 20th 2014 this indicates a bug rather >>> than a missing feature in ActiveMQ >>> * and this Spec shall pinpoint the behavior. >>> * * >>> * The test is based on ActiveMQ 5.10 >>> * >>> * Observations: >>> * ------------- >>> * Depending on when the durable subscriber is known to the members of >>> the >>> NWOB, messages can be either left pending >>> * or delivered repeatedly (see the last 2 test cases). Message gaps can >>> occur, if the DS has only connected >>> * to one broker so far. If the DS then disconnects and after a while >>> reconnects to the other broker it wasn't >>> * connected to so far, it will not see the messages that have been >>> produced while it was offline (it will see >>> * those messages after reconnecting to broker 1). >>> * >>> * Dupilcate delivery will happen if the DS was already connected to >>> both >>> brokers. From the broker's perspective >>> * it seems that those DS are handled as two distinct subscribers, so >>> effectively all messages that are published >>> * will eventually be delivered to both subscribers. >>> */ >>> >>> I know that Virtual Topics could solve the problem - however we in the >>> middleware team are not in control of that >>> particular client application and therefor we cannot change the consumer >>> from a DS to a queue consumer. >>> >>> Can you confirm that we are indeed looking at a missing feature or a bug >>> in ActiveMQ 5.10 ? - Otherwise i would >>> need to get my thinking cap back on and see how I could solve the problem >>> without changing the client code. >>> >>> [1] >>> https://github.com/woq-blended/blended/blob/master/ >>> blended-testing/blended-testing-activemq/src/test/ >>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala >>> < >>> https://github.com/woq-blended/blended/blob/master/ >>> blended-testing/blended-testing-activemq/src/test/ >>> scala/de/woq/blended/testing/activemq/DurableSubscriberSpec.scala >>> Thanks and best regards >>> Andreas >>> >>> >>> On 20 Oct 2014, at 17:40, Tim Bain <tb...@alumni.duke.edu> wrote: >>>> >>>> If you have a network of brokers, messages on topics will be forwarded >>>> to >>>> whichever broker the consumer connects to, without duplicate delivery of >>>> any messages so long as no messages were processed by the consumer >>>> >>> without >>> >>>> being ack'ed. If you were using queues, there's the potential for >>>> >>> messages >>> >>>> to get stranded on a broker if no consumers are left, but this isn't >>>> possible for topics. (I'm not clear on the reason that topics can't get >>>> messages stranded even when consumers bounce between brokers, and >>>> unfortunately http://activemq.apache.org/networks-of-brokers.html >>>> >>> doesn't >>> >>>> describe why that is.) >>>> >>>> So I think that ActiveMQ's base capabilities will do exactly what you >>>> >>> want, >>> >>>> and if you're seeing redelivery of messages that were successfully acked >>>> when the consumer bounces to another broker, I think that would indicate >>>> >>> a >>> >>>> bug in ActiveMQ rather than a missing feature. >>>> >>>> On Sun, Oct 19, 2014 at 6:05 PM, Noel OConnor <noel.ocon...@gmail.com> >>>> wrote: >>>> >>>> Take a look at idempotent consumers in camel. This may help you out as >>>>> a >>>>> basis for your plugin if you decide to go with it. >>>>> On Oct 18, 2014 5:47 PM, "Andreas Gies" <andr...@wayofquality.de> >>>>> >>>> wrote: >>> >>>> Hi >>>>>> >>>>>> I am using ActiveMQ 5.10 in an application. So far the requirement for >>>>>> >>>>> the >>>>> >>>>>> remote locations has been for pure store and forward capabilities, >>>>>> so that a single AMQ broker was sufficient. This has changed in a way >>>>>> >>>>> that >>>>> >>>>>> now 2 nodes should be present in the remote location for >>>>>> resilience and load balancing. I had considered a master/configuration >>>>>> >>>>> as >>> >>>> the requirement for resilience is stronger than that for load >>>>>> >>>>> balancing. >>> >>>> However, the situation in those locations is that I don’t have a shared >>>>>> >>>>> db >>>>> >>>>>> nor a shared filesystem. As far as I have understood the replicated >>>>>> >>>>> level db >>>>> >>>>>> would require at least 3 nodes ? >>>>>> >>>>>> This is why I have chosen a network of brokers in the end, which works >>>>>> well for any Queue based communication. >>>>>> >>>>>> Now my problem is that there is one client application that is >>>>>> provided >>>>>> >>>>> by >>>>> >>>>>> a 3rd party and uses durable subscriptions. It would be quite an >>>>>> effort >>>>>> to change that application towards using queues, so that I could >>>>>> >>>>> consider >>> >>>> virtual destinations. >>>>>> >>>>>> The problem occurs two-fold: >>>>>> >>>>>> Assume a Subscriber connects to BrokerA, then disconnects and >>>>>> >>>>> reconnects >>> >>>> to Broker B. It consumes messages for a while, than disconnects >>>>>> and reconnects to Broker A. All messages that have already been >>>>>> >>>>> consumed >>> >>>> while it was connected to Broker B will be delivered again. >>>>>> >>>>>> My question is now whether this could be avoided by means of ActiveMQ >>>>>> alone ? - I was contemplating a broker plugin to track messages that >>>>>> have been consumed on other nodes so that I could avoid redelivering >>>>>> >>>>> them >>> >>>> again. >>>>>> >>>>>> Sorry if thats a bit vague - I am fishing for ideas …. >>>>>> >>>>>> >>>>>> Thanks and best regards >>>>>> Andreas >>>>>> >>>>> >>> > -- > > > Andreas Gies > > WoQ – Way of Quality GmbH > > Geschäftsführer & CTO > > /eMail:/andr...@wayofquality.de <mailto:andr...@wayofquality.de> > > /Tel:/ +49 151 23470823 > > /Fax:/ +49 1805 006534 2114 > > /Twitter:/ andreasgies /Skype:/ giessonic > > /LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/> ( > http://de.linkedin.com/pub/andreas-gies/0/594/aa5/) > > /Xing:/ <http://www.xing.com/profile/Andreas_Gies> ( > http://www.xing.com/profile/Andreas_Gies) > > /Blog:/ <http://www.wayofquality.de/index.php/en/blog> ( > http://www.wayofquality.de/index.php/en/blog) > > /Github:/ <https://github.com/atooni> (https://github.com/atooni) > > /Amtsgericht Landshut:/HRB 8352// > > // > > /Ust.-Id.:/ DE274771254 > > > Haftungsausschluss > > Diese Email kann vertrauliche und/oder rechtlich geschützte Informationen > enthalten und ist ausschließlich für den/die benannten Adressaten bestimmt. > Sollten Sie nicht der beabsichtigte Empfänger sein oder diese Email > irrtümlich erhalten haben, ist es Ihnen nicht gestattet diese Mail oder > einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu kopieren, unbefugt > weiterzuleiten oder zu behalten. Informieren Sie bitte sofort den Absender > telefonisch oder per Email und löschen Sie diese Email und alle Kopien aus > Ihrem System. Wir haften nicht für die Unversehrtheit von Emails, nachdem > sie unseren Einflussbereich verlassen haben. > > > Disclaimer > > This email may contain confidential and/or privileged information and is > intended solely for the attention and use of the named addressee(s). If you > are not the intended recipient, or a person responsible for delivering it > to the intended recipient, you are not authorized to and must not disclose, > copy, distribute, or retain this message or any part of it without our > authority. Please contact the sender by call or reply email immediately and > destroy all copies and the original message. We are not responsible for the > integrity of emails after they have left our sphere of control. > > // >
