Consumers connect directly to individual beanstalks servers. We have our consumers just consume from 1 beanstalkd server. If a beanstalkd server is going to be down for a long time it's simple to point all of it's consumers to another beanstalkd server instead, but to be honest that's never happened.
in other words say we have bns4, bns5, bns6 beanstalkd servers and we have cns1, cns2, cns3 consumers cns1 would connect to bns4, cns2 to bns5, cns3 to bns6, etc pretty low-budget solution but works fairly well so far. -- Chad Kouse On Saturday, February 9, 2013 at 3:42 PM, News Group wrote: > How do you deal with the worker connection to the multiple beanstalk > instances? > > On Tuesday, 8 January 2013 01:24:06 UTC-5, chadkouse wrote: > > Sure, from our ops team to you: > > > > listen beanstalk-production 100.100.100.100:11300 > > (http://100.100.100.100:11300) > > mode tcp > > option abortonclose > > option log-separate-errors > > maxconn 4415 > > option httpchk > > balance roundrobin > > timeout client 120s > > timeout connect 120s > > timeout server 120s > > server bns4.shared.day1 172.0.0.4:11300 (http://172.0.0.4:11300) check port > > 9200 inter 10s fall 2 rise 5 > > server bns5.shared.day1 172.0.0.5:11300 (http://172.0.0.5:11300) check port > > 9200 inter 10s fall 2 rise 5 > > server bns6.shared.day1 172.0.0.6:11300 (http://172.0.0.6:11300) check port > > 9200 inter 10s fall 2 rise 5 > > > > for port 9200, we have a script running on each host under xinetd, that > > gives an HTTP 200 if the host is healthy, and a 304 if not. > > > > -- chad > > > > > > On Monday, January 7, 2013 at 9:00 PM, Drew Broadley wrote: > > > > > Chad, are you able to provide your haproxy configuration for beanstalkd > > > in high availability mode ? > > > > > > On Thursday, 18 October 2012 08:50:28 UTC+13, chadkouse wrote: > > > > I'm not sure about like 90% of what you said but we present our > > > > beanstalkd pool to our application as a single point of entry > > > > maintained by a set of haproxy's. works pretty well and is simple > > > > enough. > > > > > > > > -- > > > > Chad Kouse > > > > > > > > > > > > On Wednesday, October 17, 2012 at 12:43 PM, B. Estrade wrote: > > > > > > > > > I am working on a project that needs a job queue, and beanstalkd is > > > > > looking very attractive. The thing is that we really want to make the > > > > > application very highly available, and the fact that federation of > > > > > beanstalkd is left to the client (like memcached) has lead me to > > > > > consider the following approach. I'd like to create a resource > > > > > manager that maintains a pool of daemons, but presents to the client > > > > > application the view as a single and highly available process that is > > > > > managing these requests. > > > > > > > > > > I would like to basically create a resource manager that provides > > > > > failover and redundancy via a pool of distributed beanstalkd > > > > > instances. The application, however, will view things logically as a > > > > > single logical instance of the queue. It looks like that the way to > > > > > go would be to manage the pool of beanstalkd instances using Paxos. > > > > > > > > > > My primary question after reading the documentation is, how would one > > > > > use the existing features (as atomic primitives, so to speak) of > > > > > beanstalkd to create a resource manager that provides the isolation > > > > > guarantees and atomic changes to the state of a queue? As far as I > > > > > can tell, this would be possible if a) there was a way to provide a > > > > > temporary change to the queue (e.g., a job or set of jobs) and b) a > > > > > way to make the temporary changes visible in an atomic way ( I > > > > > suppose a daemon would also need to know if it's OK to make the > > > > > commit, too). I don't think that these needs are unique the to > > > > > distributed commit protocol since this would be required by each > > > > > daemon individually to facilitate the protocol itself. Leader elect, > > > > > I think, can be handled outside of the daemon. > > > > > > > > > > I would like to provide strong consistency across all beanstalkd > > > > > instances in the pool; I am not interested, really, in simply > > > > > master/slave replication (which I suppose one could facilitate via > > > > > the bin log of a master). I am interested in seeing if it is possible > > > > > to implement a Paxos like protocol among all daemons in the pool > > > > > using beanstalkd's current capabilities. Weak or eventual > > > > > consistency is also an option, but I'd like to see how possible the > > > > > ideal is at the onset. > > > > > > > > > > Thanks! > > > > > Brett > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "beanstalk-talk" group. > > > > > To view this discussion on the web visit > > > > > https://groups.google.com/d/msg/beanstalk-talk/-/wnNdSeBC_GkJ. > > > > > To post to this group, send email to [email protected]. > > > > > To unsubscribe from this group, send email to > > > > > [email protected]. > > > > > For more options, visit this group at > > > > > http://groups.google.com/group/beanstalk-talk?hl=en. > > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "beanstalk-talk" group. > > > To view this discussion on the web visit > > > https://groups.google.com/d/msg/beanstalk-talk/-/w25lHXeg7LcJ. > > > To post to this group, send email to [email protected] > > > (javascript:). > > > To unsubscribe from this group, send email to > > > [email protected] (javascript:). > > > For more options, visit this group at > > > http://groups.google.com/group/beanstalk-talk?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > (mailto:[email protected]). > To post to this group, send email to [email protected] > (mailto:[email protected]). > Visit this group at http://groups.google.com/group/beanstalk-talk?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
