This looks pretty good! When you are happy with the work, please let me know, I'll pull it back into my master.
Regards, Erich On Jan 10, 5:10 pm, mahmoud abdelkader <[email protected]> wrote: > I'm integrating proper asynchronous support into pybeanstalk, which > will be released shortly and can be followed > here:http://github.com/tjdziuba/pybeanstalk, > but I want to ask a question regarding workers, I refer to them as > clients, and their appropriate actions. > > Assuming that there's a pool of beanstalk queues, distributed across > multiple servers, is it OK to assume that if a client broadcasts a > reserve command, then all servers will receive "reserve\r\n"? Wouldn't > *ALL* queues then receive a "reserve" command which will wait > indefinitely if there are no jobs on the queue, hindering subsequent > commands? > > Imagine the following every-day scenario, where there are producers > and consumers running on different machines: > > producer: > p = Pool([Server1, Server2, Server3, Server4]) > p.put(RandomData) #only one job is placed, randomly on a server chosen > from Pool > > consumer: > c = connect_to_pool() > while True: > job = c.reserve() #issue reserve to all queues > # ... do something with the job ... > print c.stats() > > All queues, except the random recipient of the placed job, now have a > "reserve\r\n" command which hinders them from processing any more > commands, e.g. the "stats\r\n" command. > > To solve this issue, I can assume that there will be more than one job > placed, and the process of randomly choosing a server is fair and will > really distribute across multiple servers. To avoid including the > waiting servers in my randomly chosen server, I have given to them a > "waiting" attribute where, if evaluated to true, it excludes them from > the random broadcast eligibility. > > Another way, is pretty hacky, but solves the issue of small jobs, if > there are more than one server then implicitly convert all reserve\r\n > commands to have a default timeout of something small like 5, where I > can constantly poll all servers. > > Thoughts? Comments? > Mahmoud
-- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
