It seems like this could be done with the new persistence stuff in 1.4 and something like DRBD+Wackamole.
On Nov 9, 10:49 am, Pierre-Yves Kerembellec <[email protected]> wrote: > On 9 nov, 06:42, Keith Rarick <[email protected]> wrote: > > > Yes! Some coworkers discussed this with me long ago, but we never got > > a chance to do it. I don't know when I will have time to do it myself; > > if you want to implement it I will do whatever I can to help. > > > We can discuss the design here on the list. Here are some > > (underdeveloped) ideas from my old discussions: > > > * peer-peer, not master-slave > > * built-in eventual consistency > > - in a general-purpose store, the client has to resolve inconsistencies > > - but hopefully beanstalkd can resolve inconsistencies automatically > > * introduce the feature a piece at a time rather than one huge change > > * maybe embed the replication protocol in the existing protocol > > I've read some of the past discussions and I understand the different > points. The > idea I was pursuing was not to enable a full consistent cross- > replication against > multiple servers: I guess that type of scalability would be achieved > by sharding > producers and consumers on multiple beanstalkd instances, not really > talking to > each others. > > I was more thinking about some kind of "remote binlog" capability, in > a master/slave > configuration, where a "master" server would provide a stream of its > modifications to > a "slave" server, so that a catastrophic failure of the master (not at > the beanstalkd > software level but at the physical machine level) would lead to > clients switching to > the slave, with 0-to-no job loss (depending on the replication & > network speeds between > the master and the slave). > > The "switch" would be handled by some sort of floating IP mechanism > (software VRRP > like ucarpd, or quorum-based clustering like wackamole comes to mind). > This capability > is quite mandatory in our forthcoming architecture, since we heavily > rely on beanstalkd > to run multiple workflows, and we cannot afford to loose the ongoing > tasks in case > of hardware failure. > > Most of network "data" servers have some sort of master/slave > replication mechanism > to address this particular issue (MySQL, MemcacheDB, Redis, etc). I > know this is not > exactly what you had in mind (from what I've read here), but this is > an interesting > feature for us (and we are willing to implement it), so what's your > position on this ? > > Thanks, > Pierre-Yves / Dailymotion -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=.
