Instead of hopping out to a client, you could get horizontal scale and asynchronous processing by using an AsyncEventListener in the servers. That will take care of multi-threading and queuing and all the plumbing, and you just go ahead and write your processing code and deploy it as AsyncEventListeners. This gives you guaranteed ordering semantics for each key as well.
I *think* it even gives you a notion of H/A so that if the primary fails the queued messages will be processed by a secondary. (I know the WAN Gateway does and it uses pretty much the same plumbing under the covers). -- Mike Stolz Principal Engineer, GemFire Product Manager Mobile: 631-835-4771 On Thu, Jan 19, 2017 at 11:21 AM, Udo Kohlmeyer <[email protected]> wrote: > Hi there Paul, > > Firstly, your use case looks really interesting and hope to see a few more > posts on how you use Geode further. Keep us informed we like to hear what > you guys are doing with GEODE! :) > > The subscription or CQ (continuous query) paradigm is, as stated, a 1:1 > relationship. When a client registers interest on a region that client will > be notified. This is more of a topic semantic rather than a queue semantic. > > Although this is not the first time I've heard the request for this kind > of functionality. To best explain why GEODE, currently, implements the 1:1 > relationship has got to do with guaranteed delivery and in-order delivery. > If we use a queue semantic, with multiple clients being able to process > data in a balanced manner, we end up with potential out-of-order processing > of messages. In addition to that it now becomes significantly harder to > track and n deal with client failures and the potential replaying of > messages. > > But that said, I have seen other users resolve this problem and could > detail some approaches in a later correspondence if you'd like. > > --Udo > > > > On 1/19/17 03:39, Paul Perez wrote: > > Hello All, > > > > As explained in a previous email, we try to use Geode to process and > aggregate a stream of Traces. Our requirement is to process billions of > simple traces every day. > > We imagine the aggregation process in many steps. > > One: traces are generated by a tiers tools and stored in a first geode > region > > Two: once a trace put in the first region we use the async event feature > to invoke a client that executes the first aggregation steps. Then the > result will be put in a second region. > > Three: the second aggregation step is in the same way, when traces are put > in the second region, then an asynchronous event is sent to the client to > execute the second part of the aggregation etc.… > > > > For scalability purposes, we plan to use many clients that could receive > the events and execute the aggregation and put the results back to Geode. > > Consequently, as far as we understand the documentation, when an entry is > put in a region, each client that registered an interest receives an event > and aggregate the trace. So, the trace will be aggregated many times. > > > > My question is: If many clients are registered, could we configure the > region to send randomly, the event to one client only. > > A subsidiary question: Do we have the same behaviour with the function > execution feature or it could be an alternative in that case > > Thank you for your help > > Best regards > > > > Paul > > > > >
