I am working on a project that needs a job queue, and beanstalkd is looking very attractive. The thing is that we really want to make the application very highly available, and the fact that federation of beanstalkd is left to the client (like memcached) has lead me to consider the following approach. I'd like to create a resource manager that maintains a pool of daemons, but presents to the client application the view as a single and highly available process that is managing these requests.
I would like to basically create a resource manager that provides failover and redundancy via a pool of distributed beanstalkd instances. The application, however, will view things logically as a single logical instance of the queue. It looks like that the way to go would be to manage the pool of beanstalkd instances using Paxos. My primary question after reading the documentation is, how would one use the existing features (as atomic primitives, so to speak) of beanstalkd to create a resource manager that provides the isolation guarantees and atomic changes to the state of a queue? As far as I can tell, this would be possible if a) there was a way to provide a temporary change to the queue (e.g., a job or set of jobs) and b) a way to make the temporary changes visible in an atomic way ( I suppose a daemon would also need to know if it's OK to make the commit, too). I don't think that these needs are unique the to distributed commit protocol since this would be required by each daemon individually to facilitate the protocol itself. Leader elect, I think, can be handled outside of the daemon. I would like to provide strong consistency across all beanstalkd instances in the pool; I am not interested, really, in simply master/slave replication (which I suppose one could facilitate via the bin log of a master). I am interested in seeing if it is possible to implement a Paxos like protocol among all daemons in the pool using beanstalkd's current capabilities. Weak or eventual consistency is also an option, but I'd like to see how possible the ideal is at the onset. Thanks! Brett -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/wnNdSeBC_GkJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
