I'm not sure about like 90% of what you said but we present our beanstalkd pool to our application as a single point of entry maintained by a set of haproxy's. works pretty well and is simple enough.
-- Chad Kouse On Wednesday, October 17, 2012 at 12:43 PM, B. Estrade wrote: > I am working on a project that needs a job queue, and beanstalkd is looking > very attractive. The thing is that we really want to make the application > very highly available, and the fact that federation of beanstalkd is left to > the client (like memcached) has lead me to consider the following approach. > I'd like to create a resource manager that maintains a pool of daemons, but > presents to the client application the view as a single and highly available > process that is managing these requests. > > I would like to basically create a resource manager that provides failover > and redundancy via a pool of distributed beanstalkd instances. The > application, however, will view things logically as a single logical instance > of the queue. It looks like that the way to go would be to manage the pool > of beanstalkd instances using Paxos. > > My primary question after reading the documentation is, how would one use the > existing features (as atomic primitives, so to speak) of beanstalkd to create > a resource manager that provides the isolation guarantees and atomic changes > to the state of a queue? As far as I can tell, this would be possible if a) > there was a way to provide a temporary change to the queue (e.g., a job or > set of jobs) and b) a way to make the temporary changes visible in an atomic > way ( I suppose a daemon would also need to know if it's OK to make the > commit, too). I don't think that these needs are unique the to distributed > commit protocol since this would be required by each daemon individually to > facilitate the protocol itself. Leader elect, I think, can be handled > outside of the daemon. > > I would like to provide strong consistency across all beanstalkd instances in > the pool; I am not interested, really, in simply master/slave replication > (which I suppose one could facilitate via the bin log of a master). I am > interested in seeing if it is possible to implement a Paxos like protocol > among all daemons in the pool using beanstalkd's current capabilities. Weak > or eventual consistency is also an option, but I'd like to see how possible > the ideal is at the onset. > > Thanks! > Brett > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/beanstalk-talk/-/wnNdSeBC_GkJ. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
