On 9 nov, 06:42, Keith Rarick <[email protected]> wrote: > Yes! Some coworkers discussed this with me long ago, but we never got > a chance to do it. I don't know when I will have time to do it myself; > if you want to implement it I will do whatever I can to help. > > We can discuss the design here on the list. Here are some > (underdeveloped) ideas from my old discussions: > > * peer-peer, not master-slave > * built-in eventual consistency > - in a general-purpose store, the client has to resolve inconsistencies > - but hopefully beanstalkd can resolve inconsistencies automatically > * introduce the feature a piece at a time rather than one huge change > * maybe embed the replication protocol in the existing protocol
I've read some of the past discussions and I understand the different points. The idea I was pursuing was not to enable a full consistent cross- replication against multiple servers: I guess that type of scalability would be achieved by sharding producers and consumers on multiple beanstalkd instances, not really talking to each others. I was more thinking about some kind of "remote binlog" capability, in a master/slave configuration, where a "master" server would provide a stream of its modifications to a "slave" server, so that a catastrophic failure of the master (not at the beanstalkd software level but at the physical machine level) would lead to clients switching to the slave, with 0-to-no job loss (depending on the replication & network speeds between the master and the slave). The "switch" would be handled by some sort of floating IP mechanism (software VRRP like ucarpd, or quorum-based clustering like wackamole comes to mind). This capability is quite mandatory in our forthcoming architecture, since we heavily rely on beanstalkd to run multiple workflows, and we cannot afford to loose the ongoing tasks in case of hardware failure. Most of network "data" servers have some sort of master/slave replication mechanism to address this particular issue (MySQL, MemcacheDB, Redis, etc). I know this is not exactly what you had in mind (from what I've read here), but this is an interesting feature for us (and we are willing to implement it), so what's your position on this ? Thanks, Pierre-Yves / Dailymotion --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en -~----------~----~----~----~------~----~------~--~---
