The whole problem boils down to the question how the whole network of nodes is started up.
You already have your get/put/remove semantics in place. Every node can accept those commands and act on them. It sounds like a RESTfull interface. The same mechanism could be used when a node want's to go down or up. A node could send a message "put nodexyzstatus up" to other nodes. You could define some special keys which (instead of manipulating the stored data) would initiate any role changes or failover procedures you want. If a client can't reach a node, it can simply request those procedures. A "put" would initiate a change in the network and a get would be just a status query of the network. Does this make any sense? Tapio On Sun, Feb 8, 2009 at 1:24 AM, Ayende Rahien <[email protected]> wrote: > Two different things.What I described was the initial design. > The second one is the one with fail over in place. > > > On Sun, Feb 8, 2009 at 1:14 AM, Tapio Kulmala <[email protected]>wrote: > >> The nodes are NOT totally isolated from each other if they know their role >> and who's the secondary or tertiary node. I understood that the client is >> the only one who knows those things. >> >> Tapio >> >> On Sun, Feb 8, 2009 at 1:03 AM, Ayende Rahien <[email protected]> wrote: >> >>> no, you want to keep it simple >>> Ad the nodes have to know about each other, so they can tell their >>> secondary and tertiary >>> >>> >>> On Sun, Feb 8, 2009 at 1:00 AM, Tapio Kulmala <[email protected]>wrote: >>> >>>> Ok.... >>>> >>>> I'm still assuming that the client is the only one who knows what the >>>> secondary and tertiary nodes are for the primary node. >>>> >>>> If the primary node is down, could you just swap the roles of those >>>> nodes. Make the unavailable node tertiary and go on as usual? >>>> >>>> >>>> >>>> On Sun, Feb 8, 2009 at 12:53 AM, Ayende Rahien <[email protected]>wrote: >>>> >>>>> No, that is the problem that this is supposed to deal with.I am not >>>>> straight yet on the issue of how to deal with the node configuration. >>>>> >>>>> The design for the new version of Rhino DHT is simple. We continue to >>>>> support only three operations on the wire, Put, Get and Remove. But we >>>>> also >>>>> introduced a new notion. Failover servers. Every node in the DHT has a >>>>> secondary and tertiary nodes defined to it. Those nodes are also full >>>>> fledged nodes in the DHT, capable of handling their own stuff. >>>>> >>>>> During normal operation, any successful Put or Remove operation will be >>>>> sent via async messages to the secondary and tertiary nodes. If a node >>>>> goes >>>>> down, the client library is responsible for detecting that and moving to >>>>> the >>>>> secondary node, and the tertiary one if that is down as well. Get is >>>>> pretty >>>>> simple in this regard, as you can imagine, the node needs to simply serve >>>>> the request from local storage. Put and Remove operations are more >>>>> complex, >>>>> the logic for doing this is the same as always, include all the conflict >>>>> resolution, etc. But in addition to that, the Put and Remove requests will >>>>> generate async messages to the primary and tertiary nodes (if using the >>>>> secondary as fallback, and primary and secondary if using the tertiary as >>>>> fallback). >>>>> >>>>> That way, when the primary come back up, it can catch up with work that >>>>> was done while it was down. >>>>> The question of how to store the data about the nodes ,however, >>>>> remains. I think that I'll store it as replicated value in all the nodes. >>>>> So >>>>> we have a list of all the nodes and their secondary and tertiary there. >>>>> >>>>> On Sun, Feb 8, 2009 at 12:49 AM, Tapio Kulmala <[email protected] >>>>> > wrote: >>>>> >>>>>> Doesn't that algorithm always get the data for key xxx from the same >>>>>> node? If the node is down you'll hit the real data store but you'll never >>>>>> switch to another node. Or do you take the node out from your nodes list? >>>>>> >>>>>> >>>>>> Tapio >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Feb 8, 2009 at 12:35 AM, Ayende Rahien <[email protected]>wrote: >>>>>> >>>>>>> Right now, this information is stored on the client side.The client >>>>>>> has a list of nodes and get/store the data in them using the following >>>>>>> algorithm: >>>>>>> >>>>>>> public Value Get(string key) >>>>>>> { >>>>>>> return nodes[ key.GetHashCode() % nodes.Length].Get(key) >>>>>>> } >>>>>>> >>>>>>> On Sun, Feb 8, 2009 at 12:19 AM, Tapio Kulmala < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I don't know anything about Rhino DHT so this might be a really >>>>>>>> stupid question. You said that that each node is each node is totally >>>>>>>> isolated from all the rest. How does the client initially know what >>>>>>>> nodes >>>>>>>> exists and where the data might be stored? >>>>>>>> >>>>>>>> Tapio >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Feb 7, 2009 at 9:17 PM, Ayende Rahien <[email protected]>wrote: >>>>>>>> >>>>>>>>> My initial design when building Rhino DHT was that it would work >>>>>>>>> in a similar manner to Memcached, with the addition of multi versioned >>>>>>>>> values and persistence. That is, each node is completely isolated >>>>>>>>> from all >>>>>>>>> the rest, and it is the client that is actually creating the illusion >>>>>>>>> of >>>>>>>>> distributed cohesion. >>>>>>>>> >>>>>>>>> The only problem with this approach is reliability. That is, if a >>>>>>>>> node goes down, all the values that are stored in it are gone. This >>>>>>>>> is not a >>>>>>>>> problem for Memcached. If the node is down, all you have to do is to >>>>>>>>> hit the >>>>>>>>> actual data source. Memcached is *not *a data store, it is a >>>>>>>>> cache, and it is allowed to remove values when you want it. >>>>>>>>> >>>>>>>>> For Rhino DHT, that is not the case. I am using it to store the >>>>>>>>> saga details for Rhino Service Bus, as well as storing persistent >>>>>>>>> state. >>>>>>>>> >>>>>>>>> The first plan was to use it as is. If a node is down, it would >>>>>>>>> cause an error during load saga state stage (try to say *that*three >>>>>>>>> times fast!), which would eventually move the message to the error >>>>>>>>> queue, when the node came back up, we could move the messages from >>>>>>>>> the error >>>>>>>>> queue to the main queue and be done with it. >>>>>>>>> >>>>>>>>> My current client had some objections to that, from his >>>>>>>>> perspective, if any node in the DHT was down, the other nodes should >>>>>>>>> take >>>>>>>>> over automatically, without any interruption of service. That is… >>>>>>>>> somewhat >>>>>>>>> more complex to handle. >>>>>>>>> >>>>>>>>> Well, actually, it isn't more complex to handle. I was able to >>>>>>>>> continue with my current path for everything (including full >>>>>>>>> transparent >>>>>>>>> failover for reads and writes). >>>>>>>>> >>>>>>>>> What I was *not *able to solve, however, was how to handle a node >>>>>>>>> coming back *up*. Or, to be rather more exact, I run into a >>>>>>>>> problem there because the only way to solve this cleanly was to use >>>>>>>>> messaging. But, of course, Rhino Service Bus is dependent on Rhino >>>>>>>>> DHT. And >>>>>>>>> creating a circular reference would just make things more complex, >>>>>>>>> even if >>>>>>>>> it was broken with interfaces in the middle. >>>>>>>>> Thoughts? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en -~----------~----~----~----~------~----~------~--~---
