I ironed out a new PHP client for beanstalkd using much of the current client available. I have completely revamped the server selection algorithm to work as I suggested above. The beanstalkd server is selected based on the tube name provided using a consistent distributed hash table (like memcached does). Just select a tube name and the client will determine which server to look at and reserve/put on that server only. Work fabulous so far! I'll be sure to pass this along to anyone that wants it. Seems like we should easily be able to scale out to hundreds of beanstalk servers now!
- Osmaan On Jan 15, 6:04 pm, noog_com <[email protected]> wrote: > Keith, > > Thanks for the response! You are correct...sometimes I probably look > too far in the future :) The concerns you brought up are definitely > important. At the moment, we really only need one machine running > beanstalkd, but depending on the growing demand of our platform, we > may require quite a few servers to handle just beanstalkd alone. We > are migrating our entire existing message queue system to beanstalkd > right now. Will certainly let you know how things go! Will be > curious to see how well this handles as we scale. At the moment, it > certainly looks promising. > > Side note: The current php client library you have listed on your > website has a critical error on line 326 of > Beanstalk.class.php....there is a function call that is > misspelled....it says _do_my_sever, instead it should be > "_do_my_server"...took me FOREVER to figure out why use_tube wasnt > working properly :( > > Thanks again! > > - Osmaan > > On Jan 14, 5:08 pm, "Keith Rarick" <[email protected]> wrote: > > > > > On Tue, Jan 13, 2009 at 6:37 PM, noog_com <[email protected]> wrote: > > > I am looking into utilizing beanstalkd as our distributed message > > > queue system for noog.com. So far, we have been tremendously > > > impressed with the simplicity and power of this tool. > > > Thanks! And be sure to let us know about other things you think could > > be improved upon, or if you hit any trouble setting things up. > > > > Hopefully this isn't a redundant feed; however, I was wondering if > > > beanstalkd has plans to support server selection through hashing (like > > > memcached)? What I mean is, if I have 3 servers running beanstalkd > > > and I want to deliver a message into the tube called "tube_a", from > > > what I gather, the worker script will connect to each of the three > > > servers looking for jobs on "tube_a." Is this a correct assumption? > > > Typically, each worker asks one server at a time. This works out well > > if you have more workers than servers (which is typical). > > > > So if I had 100 beanstalkd servers, then it would have to look through > > > all 100 servers to find a job under "tube_a?" > > > Most current workers will pick one at random and ask it. A fully > > asynchronous worker would probably ask all of them. > > > But why would you need 100 beanstalkd servers? > > > > If the beanstalk > > > server list were hashed with the key "tube_a" then, for example, > > > server 1 would always receive the jobs for "tube_a" (again as memcache > > > does). That way the worker would immediately know to look at only > > > server 1 for "tube_a" jobs.....should be much faster...no? > > > I don't know if it would be much faster; I'm skeptical. It probably > > depends on how many workers and how many servers there are. It also > > will depend on how many different tubes you have -- it's considered > > good practice to have as few as possible. > > > You'd need to be careful to handle the case when a server goes down. > > The workers should be able to gracefully move to another server. > > > > I was thinking about expanding the current beanstalk PHP client > > > library to do this in order to speed up the job retrieval. Has anyone > > > already looked in this? What are your thoughts? > > > Give it a shot. If you can demonstrate that it's faster, I'm sure the > > client library maintainers will be interested in adopting your > > technique. > > > kr- Hide quoted text - > > - Show quoted text - --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en -~----------~----~----~----~------~----~------~--~---
