How do you deal with the worker connection to the multiple beanstalk instances?
On Tuesday, 8 January 2013 01:24:06 UTC-5, chadkouse wrote: > > Sure, from our ops team to you: > > listen beanstalk-production 100.100.100.100:11300 > mode tcp > option abortonclose > option log-separate-errors > maxconn 4415 > option httpchk > balance roundrobin > timeout client 120s > timeout connect 120s > timeout server 120s > server bns4.shared.day1 172.0.0.4:11300 check port 9200 inter 10s fall 2 > rise 5 > server bns5.shared.day1 172.0.0.5:11300 check port 9200 inter 10s fall 2 > rise 5 > server bns6.shared.day1 172.0.0.6:11300 check port 9200 inter 10s fall 2 > rise 5 > > for port 9200, we have a script running on each host under xinetd, that > gives an HTTP 200 if the host is healthy, and a 304 if not. > > -- chad > > On Monday, January 7, 2013 at 9:00 PM, Drew Broadley wrote: > > Chad, are you able to provide your haproxy configuration for beanstalkd in > high availability mode ? > > On Thursday, 18 October 2012 08:50:28 UTC+13, chadkouse wrote: > > I'm not sure about like 90% of what you said but we present our > beanstalkd pool to our application as a single point of entry maintained by > a set of haproxy's. works pretty well and is simple enough. > > -- > Chad Kouse > > On Wednesday, October 17, 2012 at 12:43 PM, B. Estrade wrote: > > I am working on a project that needs a job queue, and beanstalkd is > looking very attractive. The thing is that we really want to make the > application very highly available, and the fact that federation of > beanstalkd is left to the client (like memcached) has lead me to consider > the following approach. I'd like to create a resource manager that > maintains a pool of daemons, but presents to the client application the > view as a single and highly available process that is managing these > requests. > > I would like to basically create a resource manager that provides failover > and redundancy via a pool of distributed beanstalkd instances. The > application, however, will view things logically as a single logical > instance of the queue. It looks like that the way to go would be to manage > the pool of beanstalkd instances using Paxos. > > My primary question after reading the documentation is, how would one use > the existing features (as atomic primitives, so to speak) of beanstalkd to > create a resource manager that provides the isolation guarantees and atomic > changes to the state of a queue? As far as I can tell, this would be > possible if a) there was a way to provide a temporary change to the queue > (e.g., a job or set of jobs) and b) a way to make the temporary changes > visible in an atomic way ( I suppose a daemon would also need to know if > it's OK to make the commit, too). I don't think that these needs are > unique the to distributed commit protocol since this would be required by > each daemon individually to facilitate the protocol itself. Leader elect, > I think, can be handled outside of the daemon. > > I would like to provide strong consistency across all beanstalkd instances > in the pool; I am not interested, really, in simply master/slave > replication (which I suppose one could facilitate via the bin log of a > master). I am interested in seeing if it is possible to implement a Paxos > like protocol among all daemons in the pool using beanstalkd's current > capabilities. Weak or eventual consistency is also an option, but I'd like > to see how possible the ideal is at the onset. > > Thanks! > Brett > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/beanstalk-talk/-/wnNdSeBC_GkJ. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/beanstalk-talk/-/w25lHXeg7LcJ. > To post to this group, send email to [email protected]<javascript:> > . > To unsubscribe from this group, send email to > [email protected] <javascript:>. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
