Re: [infinispan-dev] Distributed Counter Discussion

Hi all, This is an update email. I've just made a preview for the distributed counters. You can find it in [1] In this initial version I'm targeting a more consistent approach because it covers a large uses cases (you'll lose on performance) and it's easy to test. All the updates are

Re: [infinispan-dev] Distributed Counter Discussion

On 07/04/2016 15:06, Sanne Grinovero wrote: > For the "eventually consistent" case, returning a local value might be > fine but you'd need to define also how writes are merged and what > guarantees it aims to provide (or which we want to intentionally > ignore) Fortunately a counter has

Re: [infinispan-dev] Distributed Counter Discussion

On 7 April 2016 at 13:28, Tristan Tarrant wrote: > I think the main interface definition (as it is) is fine, but it needs > to be subclassed into two different behaviours: > > - "sync": the value returned by the CF is consistent across the cluster > - "eventually consistent":

Re: [infinispan-dev] Distributed Counter Discussion

I think the main interface definition (as it is) is fine, but it needs to be subclassed into two different behaviours: - "sync": the value returned by the CF is consistent across the cluster - "eventually consistent": the value returned by the CF might be stale (maybe return its latest "localh"

Re: [infinispan-dev] Distributed Counter Discussion

Hi Pedro, there have been many questions on this thread pointing out the need to clarify the definition and which requirements it aims to solve. I don't think we can comment on the API proposal without making that clear first? It seems there was some consensus that this doesn't help for the

Re: [infinispan-dev] Distributed Counter Discussion

== Status update == Hi all, The work in progress can be found in [1]. The main interface is: CompletableFuture addAndGet(long) The CompletableFuture will allow to cover a more use cases, for example, if you need to count the number of visits, you can increment without waiting it to

Re: [infinispan-dev] Distributed Counter Discussion

On 29/03/2016 10:12, Galder Zamarreño wrote: > Good start but rather than coming out with a general design for a counter, > I'd try to start coming out with functionality needed for the most commonly > use cases for counters, e.g. > > e.g. if you're counting number of visits to your website,

Re: [infinispan-dev] Distributed Counter Discussion

Good start but rather than coming out with a general design for a counter, I'd try to start coming out with functionality needed for the most commonly use cases for counters, e.g. e.g. if you're counting number of visits to your website, you only need an incrementing counter (and maybe a reset

Re: [infinispan-dev] Distributed Counter Discussion

On Tue, Mar 22, 2016 at 7:12 PM, Bela Ban wrote: > > > On 22/03/16 18:04, Dan Berindei wrote: >> On Mon, Mar 21, 2016 at 1:43 PM, Bela Ban wrote: >>> >>> >>> On 21/03/16 11:12, Pedro Ruivo wrote: Hi all, @Eric, thanks for the requirements.

Re: [infinispan-dev] Distributed Counter Discussion

On 22/03/16 18:04, Dan Berindei wrote: > On Mon, Mar 21, 2016 at 1:43 PM, Bela Ban wrote: >> >> >> On 21/03/16 11:12, Pedro Ruivo wrote: >>> Hi all, >>> >>> @Eric, thanks for the requirements. >>> >>> @Bela, does JGroups counter supports that semantics (AP)? >> >> No. You'd

Re: [infinispan-dev] Distributed Counter Discussion

On Mon, Mar 21, 2016 at 1:43 PM, Bela Ban wrote: > > > On 21/03/16 11:12, Pedro Ruivo wrote: >> Hi all, >> >> @Eric, thanks for the requirements. >> >> @Bela, does JGroups counter supports that semantics (AP)? > > No. You'd have to catch the MergeView and do this manually. I

Re: [infinispan-dev] Distributed Counter Discussion

Hi all, @Eric, thanks for the requirements. @Bela, does JGroups counter supports that semantics (AP)? Infinispan does not have eventually consistency (yet) neither an update log. So, it can't reconcile the counter and you will lose one of the partition updates. On 03/18/2016 02:19 PM, Eric

Re: [infinispan-dev] Distributed Counter Discussion

Greetings. Apologies for coming in a bit late on this conversation. Tristan pointed me to it a couple of days ago and unfortunately I'm just now getting time to reply. I can try to quickly give an overview of apiman's (JBoss API Management Gateway) requirements. What we're trying to do is

Re: [infinispan-dev] Distributed Counter Discussion

So actually you don't care if you have multiple counters in case of a network split, but you do care that the numbers of different counters get reconciled when a network partition heals. Example - C1: 1000 - Network split: C1: 1000, C2: 1000 - Different clients update counters on both sides of

Re: [infinispan-dev] Distributed Counter Discussion

Stupid question: whay do you need a distributed counter for this? Is the service you're monitoring replicated in a cluster? On 17/03/16 18:06, Eric Wittmann wrote: > Greetings. Apologies for coming in a bit late on this conversation. > Tristan pointed me to it a couple of days ago and

Re: [infinispan-dev] Distributed Counter Discussion

Yes, precisely. The API Gateway itself is clustered. It services a large volume of inbound traffic which it reverse-proxies to appropriate back-end APIs after applying policies such as security, rate limiting, caching, etc. -Eric On 3/18/2016 2:32 AM, Bela Ban wrote: > Stupid question: whay

Re: [infinispan-dev] Distributed Counter Discussion

Agreed. :) On 3/18/2016 9:31 AM, Bela Ban wrote: > So actually you don't care if you have multiple counters in case of a > network split, but you do care that the numbers of different counters > get reconciled when a network partition heals. > > Example > - C1: 1000 > - Network split: C1: 1000,

Re: [infinispan-dev] Distributed Counter Discussion

> On Mar 15, 2016, at 2:12 AM, Bela Ban wrote: > > > > On 14/03/16 23:17, Randall Hauch wrote: >> What are the requirements? What are the distributed counters for? Is the >> counter to be monotonically increasing? Can there be any missed values? >> Does the counter need to

Re: [infinispan-dev] Distributed Counter Discussion

> On Mar 15, 2016, at 3:26 AM, Radim Vansa wrote: > > I second Sanne's opinion about the sync/async API: make the API express > the synchronicity directly. I would even propose that for synchronous > methods, there's only the CompletableFuture or > CompletableFuture

Re: [infinispan-dev] Distributed Counter Discussion

On 03/15/2016 12:58 PM, Pedro Ruivo wrote: > > On 03/14/2016 09:27 PM, Sanne Grinovero wrote: >> Rather than a split organized on the backing implementation details, >> I'd prefer to see a split based on purpose of the counter. > So, what about multiple CounterManager where in each implementation

Re: [infinispan-dev] Distributed Counter Discussion

Comments inline On 03/14/2016 09:27 PM, Sanne Grinovero wrote: > On 14 March 2016 at 19:14, Pedro Ruivo wrote: > > As a user, how do I get a Counter instance? From a the CacheContainer > interface? You can get a Counter from a CounterManager. The CounterManager is

Re: [infinispan-dev] Distributed Counter Discussion

> For information about CRDTs, make sure you’ve read the CRDT paper by > Shapiro: http://hal.upmc.fr/inria-0088/document > On that topic, I recently came across the paper "Scalable Eventually Consistent Counters over Unreliable Networks" [1] which highlights some of the scalability problems

Re: [infinispan-dev] Distributed Counter Discussion

I second Sanne's opinion about the sync/async API: make the API express the synchronicity directly. I would even propose that for synchronous methods, there's only the CompletableFuture or CompletableFuture variant; we are already reaching the concurrency limits of application that lets

Re: [infinispan-dev] Distributed Counter Discussion

On 15/03/2016 08:00, Bela Ban wrote: > What are the requirements? To offer a cluster wide counter from an > Infinispan cache? Or is this going to be used internally, by Infinispan? The starting point is a requirement by apiman as a way to manage throttling/quotas. Currently their requirement is

Re: [infinispan-dev] Distributed Counter Discussion

On 14/03/16 23:17, Randall Hauch wrote: > What are the requirements? What are the distributed counters for? Is the > counter to be monotonically increasing? Can there be any missed values? > Does the counter need to increment and decrement? What is the *smallest* > API you need initially? > >

Re: [infinispan-dev] Distributed Counter Discussion

What are the requirements? To offer a cluster wide counter from an Infinispan cache? Or is this going to be used internally, by Infinispan? I've implemented 2 counters, in JGroups and in jgroups-raft. Here are their properties: #1 JGroups counter: - Cluster wide _named_ counters with atomicity

Re: [infinispan-dev] Distributed Counter Discussion

What are the requirements? What are the distributed counters for? Is the counter to be monotonically increasing? Can there be any missed values? Does the counter need to increment and decrement? What is the *smallest* API you need initially? There are two choices when implementing a

Re: [infinispan-dev] Distributed Counter Discussion

Great starting point! Some comments inline: On 14 March 2016 at 19:14, Pedro Ruivo wrote: > Hi everybody, > > Discussion about distributed counters. > > == Public API == > > interface Counter As a user, how do I get a Counter instance? From a the CacheContainer interface?

[infinispan-dev] Distributed Counter Discussion

Hi everybody, Discussion about distributed counters. == Public API == interface Counter String getName() //counter name long get() //current value. may return stale value due to concurrent operations to other nodes. void increment() //async or sync increment. default add(1) void