On Wednesday, October 17, 2012 3:17:38 PM UTC-5, Dave Gardner wrote: > > Highly available and redundant (and paxos) suggests to me that you have to > write to more than one queue. What are you planning to do on consumption? >
This is what I have to figure out, i.e., are there a series of things I can do to provide the isolation (temporary add to all), then the atomic switch to make it visible (durable) upon distributed consensus being met. Basically, I want to have a single logical queue that is supported by a set of many queues that can fail or go away, but the logical queue is always there and consistent. Thanks, Brett > > Sent from my iPhone > > On 17 Oct 2012, at 20:50, Chad Kouse <[email protected] <javascript:>> > wrote: > > I'm not sure about like 90% of what you said but we present our > beanstalkd pool to our application as a single point of entry maintained by > a set of haproxy's. works pretty well and is simple enough. > > -- > Chad Kouse > > On Wednesday, October 17, 2012 at 12:43 PM, B. Estrade wrote: > > I am working on a project that needs a job queue, and beanstalkd is > looking very attractive. The thing is that we really want to make the > application very highly available, and the fact that federation of > beanstalkd is left to the client (like memcached) has lead me to consider > the following approach. I'd like to create a resource manager that > maintains a pool of daemons, but presents to the client application the > view as a single and highly available process that is managing these > requests. > > I would like to basically create a resource manager that provides failover > and redundancy via a pool of distributed beanstalkd instances. The > application, however, will view things logically as a single logical > instance of the queue. It looks like that the way to go would be to manage > the pool of beanstalkd instances using Paxos. > > My primary question after reading the documentation is, how would one use > the existing features (as atomic primitives, so to speak) of beanstalkd to > create a resource manager that provides the isolation guarantees and atomic > changes to the state of a queue? As far as I can tell, this would be > possible if a) there was a way to provide a temporary change to the queue > (e.g., a job or set of jobs) and b) a way to make the temporary changes > visible in an atomic way ( I suppose a daemon would also need to know if > it's OK to make the commit, too). I don't think that these needs are > unique the to distributed commit protocol since this would be required by > each daemon individually to facilitate the protocol itself. Leader elect, > I think, can be handled outside of the daemon. > > I would like to provide strong consistency across all beanstalkd instances > in the pool; I am not interested, really, in simply master/slave > replication (which I suppose one could facilitate via the bin log of a > master). I am interested in seeing if it is possible to implement a Paxos > like protocol among all daemons in the pool using beanstalkd's current > capabilities. Weak or eventual consistency is also an option, but I'd like > to see how possible the ideal is at the onset. > > Thanks! > Brett > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/beanstalk-talk/-/wnNdSeBC_GkJ. > To post to this group, send email to [email protected]<javascript:> > . > To unsubscribe from this group, send email to > [email protected] <javascript:>. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To post to this group, send email to [email protected]<javascript:> > . > To unsubscribe from this group, send email to > [email protected] <javascript:>. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/ADiHVMTZLZcJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
