Hi Siano, the moment you want to run Camel in a distributed environment, you have to be aware that there are a number EIPs which have local state: LoabBalancers, Resequencer, Sampler, Throttler, Delayer. Non of these patterns allow you to expose the internal state, so it is not possible to use a distributed cache. This is something we have to improve in Camel and allow the users to distribute the state if required.
What you can do, is depending on your use case, identify a sensible workaround. You can have some kind of Singleton Camel route/consumer that is still HA, using things like: Camel ZooKeeper Component, Camel JGroups Component, JBoss Fuse Master Component, Database Route Policy, ActiveMQ exclusing consumers, etc. I have described the limitation of stateful EIPs and the possible workarounds as listed above in my Camel Design Patterns book [1]. It spans couple of chapters: ServiceInstancePattern, SingletonServicePattern, ThrottlingPattern (is also related). [1] https://leanpub.com/camel-design-patterns HTH, Bilgin On 13 May 2016 at 06:41, Siano, Stephan <[email protected]> wrote: > Thanks for the input. We already thought about exclusive consumers for this > scenarios (but this will essentially mean that the whole end-to-end scenario > does not scale, so we were looking for alternatives where only parts of the > scenario do not scale). The zookeeper thing is a useful hint, I wasn't aware > of this (though setting up a zookeeper cluster just for this use case might > be a bit too much overhead). > > Best regards > Stephan > > -----Original Message----- > From: Quinn Stevenson [mailto:[email protected]] > Sent: Donnerstag, 12. Mai 2016 16:47 > To: [email protected] > Subject: Re: Resequencer in distributed environment > > I had a similar situation with one implementation - we weren’t using the > resequencer, but we required FIFO processing and we wanted to have the > consumers/processers distributed across multiple containers. Since we > required FIFO, only one of the deployed components could be processing at any > given time. > > I tried both the ZooKeeperRoutePolicy ( > http://camel.apache.org/zookeeper.html > <http://camel.apache.org/zookeeper.html> ) and ActiveMQ Exclusive Consumers ( > http://activemq.apache.org/exclusive-consumer.html > <http://activemq.apache.org/exclusive-consumer.html> ). Both techniques > worked, but we were already using ActiveMQ as the message source and I didn’t > have a readily available ZooKeeper cluster in production, so I picked the > exclusive consumers. > > For what it’s worth ... > >> On May 12, 2016, at 4:54 AM, Siano, Stephan <[email protected]> wrote: >> >> Hi Claus, >> >> Thanks, that's what I also guessed from the code. Theoretically there are >> two approaches how something like that could work in a distributed >> environment: distribute the state somehow (with some replicated cache or >> something like that) or limit the execution to one node at a time (with >> route policies?). >> >> Has anybody ever tried something like that (with or without success)? >> >> Best regards >> Stephan >> >> -----Original Message----- >> From: Claus Ibsen [mailto:[email protected]] >> Sent: Donnerstag, 12. Mai 2016 11:06 >> To: [email protected] >> Subject: Re: Resequencer in distributed environment >> >> Its not distributed, and its in-memory only for a single camel context. >> >> On Thu, May 12, 2016 at 9:33 AM, Siano, Stephan <[email protected]> >> wrote: >>> Hi, >>> >>> has anybody ever tried to use a resequencer in a distributed environment >>> (e.g. with two cluster nodes doing message processing)? I am well aware >>> that the sequencer pattern as such does not scale, but it might be >>> necessary to have multiple cluster nodes for availability reasons or to >>> scale out with other processing steps that are executed before the >>> resequencer. >>> >>> From what I have found in the code the current resequencer implementation >>> relies on running on instance, but the documentation does not say anything >>> about it. >>> >>> Am I correct with my assumptions or did I miss something (e.g. some >>> distributed sequencer implementation in some camel component)? >>> >>> Has anybody successfully used the resequencer in a HA scenario (and can >>> give some hints about the approach used for that)? >>> >>> Best regards >>> Stephan >> >> >> >> -- >> Claus Ibsen >> ----------------- >> http://davsclaus.com @davsclaus >> Camel in Action 2: https://www.manning.com/ibsen2 > -- Bilgin Ibryam Camel Committer at ASF & Integration Architect at Red Hat Blog: http://ofbizian.com | Twitter: @bibryam Camel Design Patterns https://leanpub.com/camel-design-patterns Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475
