Agreed. Maybe the simplest solution is to use the service bus and break the dependency from SB to DHT. From the SB's point of view DHT is only (?) a data storage. Break that hard dependency and wire things up with an IoC.
Tapio On Sun, Feb 8, 2009 at 10:16 AM, Ayende Rahien <[email protected]> wrote: > That makes sense, but it seems to include high additional complexity that > doesn't exists in a staticly defined systemI expand on that idea here: > Rhino DHT and failover and replication, on > my!<http://ayende.com/Blog/archive/2009/02/08/rhino-dht-and-failover-and-replication-on-my.aspx> > > > On Sun, Feb 8, 2009 at 10:10 AM, Tapio Kulmala <[email protected]>wrote: > >> The whole problem boils down to the question how the whole network of >> nodes is started up. >> >> You already have your get/put/remove semantics in place. Every node can >> accept those commands and act on them. It sounds like a RESTfull interface. >> The same mechanism could be used when a node want's to go down or up. A node >> could send a message "put nodexyzstatus up" to other nodes. You could define >> some special keys which (instead of manipulating the stored data) would >> initiate any role changes or failover procedures you want. If a client can't >> reach a node, it can simply request those procedures. A "put" would initiate >> a change in the network and a get would be just a status query of the >> network. >> >> Does this make any sense? >> >> Tapio >> >> >> On Sun, Feb 8, 2009 at 1:24 AM, Ayende Rahien <[email protected]> wrote: >> >>> Two different things.What I described was the initial design. >>> The second one is the one with fail over in place. >>> >>> >>> On Sun, Feb 8, 2009 at 1:14 AM, Tapio Kulmala <[email protected]>wrote: >>> >>>> The nodes are NOT totally isolated from each other if they know their >>>> role and who's the secondary or tertiary node. I understood that the client >>>> is the only one who knows those things. >>>> >>>> Tapio >>>> >>>> On Sun, Feb 8, 2009 at 1:03 AM, Ayende Rahien <[email protected]>wrote: >>>> >>>>> no, you want to keep it simple >>>>> Ad the nodes have to know about each other, so they can tell their >>>>> secondary and tertiary >>>>> >>>>> >>>>> On Sun, Feb 8, 2009 at 1:00 AM, Tapio Kulmala >>>>> <[email protected]>wrote: >>>>> >>>>>> Ok.... >>>>>> >>>>>> I'm still assuming that the client is the only one who knows what the >>>>>> secondary and tertiary nodes are for the primary node. >>>>>> >>>>>> If the primary node is down, could you just swap the roles of those >>>>>> nodes. Make the unavailable node tertiary and go on as usual? >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Feb 8, 2009 at 12:53 AM, Ayende Rahien <[email protected]>wrote: >>>>>> >>>>>>> No, that is the problem that this is supposed to deal with.I am not >>>>>>> straight yet on the issue of how to deal with the node configuration. >>>>>>> >>>>>>> The design for the new version of Rhino DHT is simple. We continue to >>>>>>> support only three operations on the wire, Put, Get and Remove. But we >>>>>>> also >>>>>>> introduced a new notion. Failover servers. Every node in the DHT has a >>>>>>> secondary and tertiary nodes defined to it. Those nodes are also full >>>>>>> fledged nodes in the DHT, capable of handling their own stuff. >>>>>>> >>>>>>> During normal operation, any successful Put or Remove operation will >>>>>>> be sent via async messages to the secondary and tertiary nodes. If a >>>>>>> node >>>>>>> goes down, the client library is responsible for detecting that and >>>>>>> moving >>>>>>> to the secondary node, and the tertiary one if that is down as well. >>>>>>> Get is >>>>>>> pretty simple in this regard, as you can imagine, the node needs to >>>>>>> simply >>>>>>> serve the request from local storage. Put and Remove operations are more >>>>>>> complex, the logic for doing this is the same as always, include all the >>>>>>> conflict resolution, etc. But in addition to that, the Put and Remove >>>>>>> requests will generate async messages to the primary and tertiary nodes >>>>>>> (if >>>>>>> using the secondary as fallback, and primary and secondary if using the >>>>>>> tertiary as fallback). >>>>>>> >>>>>>> That way, when the primary come back up, it can catch up with work >>>>>>> that was done while it was down. >>>>>>> The question of how to store the data about the nodes ,however, >>>>>>> remains. I think that I'll store it as replicated value in all the >>>>>>> nodes. So >>>>>>> we have a list of all the nodes and their secondary and tertiary there. >>>>>>> >>>>>>> On Sun, Feb 8, 2009 at 12:49 AM, Tapio Kulmala < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Doesn't that algorithm always get the data for key xxx from the same >>>>>>>> node? If the node is down you'll hit the real data store but you'll >>>>>>>> never >>>>>>>> switch to another node. Or do you take the node out from your nodes >>>>>>>> list? >>>>>>>> >>>>>>>> >>>>>>>> Tapio >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Feb 8, 2009 at 12:35 AM, Ayende Rahien >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>>> Right now, this information is stored on the client side.The >>>>>>>>> client has a list of nodes and get/store the data in them using the >>>>>>>>> following algorithm: >>>>>>>>> >>>>>>>>> public Value Get(string key) >>>>>>>>> { >>>>>>>>> return nodes[ key.GetHashCode() % nodes.Length].Get(key) >>>>>>>>> } >>>>>>>>> >>>>>>>>> On Sun, Feb 8, 2009 at 12:19 AM, Tapio Kulmala < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I don't know anything about Rhino DHT so this might be a really >>>>>>>>>> stupid question. You said that that each node is each node is totally >>>>>>>>>> isolated from all the rest. How does the client initially know what >>>>>>>>>> nodes >>>>>>>>>> exists and where the data might be stored? >>>>>>>>>> >>>>>>>>>> Tapio >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Feb 7, 2009 at 9:17 PM, Ayende Rahien >>>>>>>>>> <[email protected]>wrote: >>>>>>>>>> >>>>>>>>>>> My initial design when building Rhino DHT was that it would work >>>>>>>>>>> in a similar manner to Memcached, with the addition of multi >>>>>>>>>>> versioned >>>>>>>>>>> values and persistence. That is, each node is completely isolated >>>>>>>>>>> from all >>>>>>>>>>> the rest, and it is the client that is actually creating the >>>>>>>>>>> illusion of >>>>>>>>>>> distributed cohesion. >>>>>>>>>>> >>>>>>>>>>> The only problem with this approach is reliability. That is, if a >>>>>>>>>>> node goes down, all the values that are stored in it are gone. This >>>>>>>>>>> is not a >>>>>>>>>>> problem for Memcached. If the node is down, all you have to do is >>>>>>>>>>> to hit the >>>>>>>>>>> actual data source. Memcached is *not *a data store, it is a >>>>>>>>>>> cache, and it is allowed to remove values when you want it. >>>>>>>>>>> >>>>>>>>>>> For Rhino DHT, that is not the case. I am using it to store the >>>>>>>>>>> saga details for Rhino Service Bus, as well as storing persistent >>>>>>>>>>> state. >>>>>>>>>>> >>>>>>>>>>> The first plan was to use it as is. If a node is down, it would >>>>>>>>>>> cause an error during load saga state stage (try to say *that*three >>>>>>>>>>> times fast!), which would eventually move the message to the error >>>>>>>>>>> queue, when the node came back up, we could move the messages from >>>>>>>>>>> the error >>>>>>>>>>> queue to the main queue and be done with it. >>>>>>>>>>> >>>>>>>>>>> My current client had some objections to that, from his >>>>>>>>>>> perspective, if any node in the DHT was down, the other nodes >>>>>>>>>>> should take >>>>>>>>>>> over automatically, without any interruption of service. That is… >>>>>>>>>>> somewhat >>>>>>>>>>> more complex to handle. >>>>>>>>>>> >>>>>>>>>>> Well, actually, it isn't more complex to handle. I was able to >>>>>>>>>>> continue with my current path for everything (including full >>>>>>>>>>> transparent >>>>>>>>>>> failover for reads and writes). >>>>>>>>>>> >>>>>>>>>>> What I was *not *able to solve, however, was how to handle a >>>>>>>>>>> node coming back *up*. Or, to be rather more exact, I run into a >>>>>>>>>>> problem there because the only way to solve this cleanly was to use >>>>>>>>>>> messaging. But, of course, Rhino Service Bus is dependent on Rhino >>>>>>>>>>> DHT. And >>>>>>>>>>> creating a circular reference would just make things more complex, >>>>>>>>>>> even if >>>>>>>>>>> it was broken with interfaces in the middle. >>>>>>>>>>> Thoughts? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Rhino Tools Dev" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/rhino-tools-dev?hl=en -~----------~----~----~----~------~----~------~--~---
