Re: Sequence Number Generation With Zookeeper

2010-08-11 Thread Adam Rosien
What happens during a network partition and different clients are incrementing different counters, and then the partition goes away? Won't (potentially) the same sequence value be given out to two clients? .. Adam On Thu, Aug 5, 2010 at 5:38 PM, Jonathan Holloway jonathan.hollo...@gmail.com

Re: Sequence Number Generation With Zookeeper

2010-08-11 Thread Ted Dunning
Can't happen. In a network partition, the side without a quorum can't update the file version. On Wed, Aug 11, 2010 at 3:11 PM, Adam Rosien a...@rosien.net wrote: What happens during a network partition and different clients are incrementing different counters, and then the partition goes

Re: Sequence Number Generation With Zookeeper

2010-08-11 Thread Adam Rosien
Ah thanks, I forgot the majority-commit property because I also forgot that all servers know what the cluster should look like, rather than act adaptively (which wouldn't make sense after all). .. Adam On Wed, Aug 11, 2010 at 3:23 PM, Ted Dunning ted.dunn...@gmail.com wrote: Can't happen. In

Re: Sequence Number Generation With Zookeeper

2010-08-10 Thread David Rosenstrauch
Good news! I got approval to release this code! (Man, I love working for a startup!!!) :-) So anyone know: what's the next step? Do I need to obtain commit privileges? Or do I deliver the code to someone who has commit privs who shepherds this for me? Also, what (if anything) do I

Re: Sequence Number Generation With Zookeeper

2010-08-10 Thread Patrick Hunt
Great! Basic details are here (create a jira, attach a patch, click submit and someone will review and help you get it into a state which we can commit). Probably you'd put your code into src/recipes or src/contrib (recipes sounds reasonable).

Re: Sequence Number Generation With Zookeeper

2010-08-10 Thread David Rosenstrauch
OK, will do, as soon as time permits. It'll take me a little while to do the needed tweaks. (Plus I'm under some pretty heavy deadline on some other work right now.) Will email back once I've got this done. DR On 08/10/2010 01:10 PM, Patrick Hunt wrote: Great! Basic details are here

Re: Sequence Number Generation With Zookeeper

2010-08-07 Thread Me
Hi all, we have something implementing the optimistic concurrency approach to sequence generation that we've been running in production for some time now. We don't see a huge amount of contention over the sequence counters as the nature of our app lends itself well to partitioned keys. Initially,

Re: Sequence Number Generation With Zookeeper

2010-08-06 Thread David Rosenstrauch
Perhaps. I'd have to ask my boss for permission to release the code. Is this something that would be interesting/useful to other people? If so, I can ask about it. DR On 08/05/2010 11:02 PM, Jonathan Holloway wrote: Hi David, We did discuss potentially doing this as well. It would be

Re: Sequence Number Generation With Zookeeper

2010-08-06 Thread Mahadev Konar
Hi David, I think it would be really useful. It would be very helpful for someone looking for geenrating unique tokens/generations ids ( I can think of plenty of applications for this). Please do consider contributing it back to the community! Thanks mahadev On 8/6/10 7:10 AM, David

Re: Sequence Number Generation With Zookeeper

2010-08-06 Thread David Rosenstrauch
I'll run it by my boss next week. DR On 08/06/2010 07:30 PM, Mahadev Konar wrote: Hi David, I think it would be really useful. It would be very helpful for someone looking for geenrating unique tokens/generations ids ( I can think of plenty of applications for this). Please do consider

Re: Sequence Number Generation With Zookeeper

2010-08-06 Thread Ted Dunning
Tell him that we will all look over your code so he gets immediate free consulting. On Fri, Aug 6, 2010 at 7:39 PM, David Rosenstrauch dar...@darose.netwrote: I'll run it by my boss next week. DR On 08/06/2010 07:30 PM, Mahadev Konar wrote: Hi David, I think it would be really useful.

Re: Sequence Number Generation With Zookeeper

2010-08-05 Thread Ted Dunning
(b) BUT: Sequential numbering is a special case of now. In large diameters, now gets very expensive. This is a special case of that assertion. If there is a way to get away from this presumption of the need for sequential numbering, you will be miles better off. HOWEVER: ZK can do better

Re: Sequence Number Generation With Zookeeper

2010-08-05 Thread Jonathan Holloway
Hi Ted, Thanks for the comments. I might have overlooked something here, but is it also possible to do the following: 1. Create a PERSISTENT node 2. Have multiple clients set the data on the node, e.g. Stat stat = zookeeper.setData(SEQUENCE, ArrayUtils.EMPTY_BYTE_ARRAY, -1); 3. Use the version

Re: Sequence Number Generation With Zookeeper

2010-08-05 Thread Ted Dunning
Sounds right to me. Much simpler as well. On Thu, Aug 5, 2010 at 5:38 PM, Jonathan Holloway jonathan.hollo...@gmail.com wrote: Hi Ted, Thanks for the comments. I might have overlooked something here, but is it also possible to do the following: 1. Create a PERSISTENT node 2. Have

Re: Sequence Number Generation With Zookeeper

2010-08-05 Thread David Rosenstrauch
On 08/05/2010 06:31 PM, Jonathan Holloway wrote: Hi all, I'm looking at using Zookeeper for distributed sequence number generation. What's the best way to do this currently? Is there a particular recipe available for this? My so far involve: a) Creating a node with PERSISTENT_SEQUENTIAL

Re: Sequence Number Generation With Zookeeper

2010-08-05 Thread Jonathan Holloway
Great, I just need to overcome the Integer.MAX_VALUE bit, maybe by having SEQUENCE_LOW and SEQUENCE_HIGH nodes, being careful with the rollover. Thanks again. On 5 August 2010 17:54, Ted Dunning ted.dunn...@gmail.com wrote: Sounds right to me. Much simpler as well. On Thu, Aug 5, 2010 at

Re: Sequence Number Generation With Zookeeper

2010-08-05 Thread Jonathan Holloway
Hi David, We did discuss potentially doing this as well. It would be nice to get some recipes for Zookeeper done for this area, if people think it's useful. Were you thinking of submitting this back as a recipe, if not then I could potentially work on such a recipe instead. Many thanks, Jon.