Hi all, we have something implementing the optimistic concurrency approach to sequence generation that we've been running in production for some time now. We don't see a huge amount of contention over the sequence counters as the nature of our app lends itself well to partitioned keys. Initially, we coded up the simplest thing we thought could work and deployed it, figuring that we'd have plenty of scope for improvement once we saw it running with real load. However, to date its been ticking over so well we've not really had cause to spend any further effort on it.
There's plenty of scope for improvement though, two of the things we had thought we would need to do sooner rather than later are implement an exponential backoff scheme (like Ted describes) when there is contention over a given counter, and to add a more performant network interface than HTTP. Like I say though, this just hasn't been a high enough priority for us yet. Anyway, we've been meaning to open source this for a while now, and prompted by this thread, I just spent an afternoon tidying up a little and pushing to github. Its at http://github.com/talisplatform/H1 and any feedback would be gratefully received. Cheers, Sam On 7 August 2010 03:40, Ted Dunning <ted.dunn...@gmail.com> wrote: > Tell him that we will all look over your code so he gets immediate free > consulting. > > On Fri, Aug 6, 2010 at 7:39 PM, David Rosenstrauch <dar...@darose.net > >wrote: > > > I'll run it by my boss next week. > > > > DR > > > > > > On 08/06/2010 07:30 PM, Mahadev Konar wrote: > > > >> Hi David, > >> I think it would be really useful. It would be very helpful for someone > >> looking for geenrating unique tokens/generations ids ( I can think of > >> plenty > >> of applications for this). > >> > >> Please do consider contributing it back to the community! > >> > >> Thanks > >> mahadev > >> > >> > >> On 8/6/10 7:10 AM, "David Rosenstrauch"<dar...@darose.net> wrote: > >> > >> Perhaps. I'd have to ask my boss for permission to release the code. > >>> > >>> Is this something that would be interesting/useful to other people? If > >>> so, I can ask about it. > >>> > >>> DR > >>> > >>> On 08/05/2010 11:02 PM, Jonathan Holloway wrote: > >>> > >>>> Hi David, > >>>> > >>>> We did discuss potentially doing this as well. It would be nice to > get > >>>> some > >>>> recipes for Zookeeper done for this area, if people think it's useful. > >>>> Were > >>>> you thinking of submitting this back as a recipe, if not then I could > >>>> potentially work on such a recipe instead. > >>>> > >>>> Many thanks, > >>>> Jon. > >>>> > >>>> > >>>> I just ran into this exact situation, and handled it like so: > >>>>> > >>>>> I wrote a library that uses the option (b) you described above. Only > >>>>> instead of requesting a single sequence number, you request a block > of > >>>>> them > >>>>> at a time from Zookeeper, and then locally use them up one by one > from > >>>>> the > >>>>> block you retrieved. Retrieving by block (e.g., by blocks of 10000 > at > >>>>> a > >>>>> time) eliminates the contention issue. > >>>>> > >>>>> Then, if you're finished assigning ID's from that block, but still > have > >>>>> a > >>>>> bunch of ID's left in the block, the library has another function to > >>>>> "push > >>>>> back" the unused ID's. They'll then get pulled again in the next > block > >>>>> retrieval. > >>>>> > >>>>> We don't actually have this code running in production yet, so I > can't > >>>>> vouch for how well it works. But the design was reviewed and given > the > >>>>> thumbs up by the core developers on the team, and the implementation > >>>>> passes > >>>>> all my unit tests. > >>>>> > >>>>> HTH. Feel free to email back with specific questions if you'd like > >>>>> more > >>>>> details. > >>>>> > >>>>> DR > >>>>> > >>>> >
