Highly available and redundant (and paxos) suggests to me that you have to write to more than one queue. What are you planning to do on consumption?
Sent from my iPhone On 17 Oct 2012, at 20:50, Chad Kouse <[email protected]> wrote: > I'm not sure about like 90% of what you said but we present our beanstalkd > pool to our application as a single point of entry maintained by a set of > haproxy's. works pretty well and is simple enough. > > -- > Chad Kouse > > On Wednesday, October 17, 2012 at 12:43 PM, B. Estrade wrote: > >> I am working on a project that needs a job queue, and beanstalkd is looking >> very attractive. The thing is that we really want to make the application >> very highly available, and the fact that federation of beanstalkd is left to >> the client (like memcached) has lead me to consider the following approach. >> I'd like to create a resource manager that maintains a pool of daemons, but >> presents to the client application the view as a single and highly available >> process that is managing these requests. >> >> I would like to basically create a resource manager that provides failover >> and redundancy via a pool of distributed beanstalkd instances. The >> application, however, will view things logically as a single logical >> instance of the queue. It looks like that the way to go would be to manage >> the pool of beanstalkd instances using Paxos. >> >> My primary question after reading the documentation is, how would one use >> the existing features (as atomic primitives, so to speak) of beanstalkd to >> create a resource manager that provides the isolation guarantees and atomic >> changes to the state of a queue? As far as I can tell, this would be >> possible if a) there was a way to provide a temporary change to the queue >> (e.g., a job or set of jobs) and b) a way to make the temporary changes >> visible in an atomic way ( I suppose a daemon would also need to know if >> it's OK to make the commit, too). I don't think that these needs are unique >> the to distributed commit protocol since this would be required by each >> daemon individually to facilitate the protocol itself. Leader elect, I >> think, can be handled outside of the daemon. >> >> I would like to provide strong consistency across all beanstalkd instances >> in the pool; I am not interested, really, in simply master/slave replication >> (which I suppose one could facilitate via the bin log of a master). I am >> interested in seeing if it is possible to implement a Paxos like protocol >> among all daemons in the pool using beanstalkd's current capabilities. Weak >> or eventual consistency is also an option, but I'd like to see how possible >> the ideal is at the onset. >> >> Thanks! >> Brett >> -- >> You received this message because you are subscribed to the Google Groups >> "beanstalk-talk" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/beanstalk-talk/-/wnNdSeBC_GkJ. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/beanstalk-talk?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
