We have been using beanstalk as a work queue for several years, and have been able to maintain a low enough load of ready jobs for this not to be a concern, but recently we have encountered some unexpected behavior: With a lot of jobs in the ready state, all with the same priority, new jobs, or newly ready jobs will be picked up before older existing jobs.
I dug into the beanstalk code, and discovered that this is due to the heapremove process. Please correct me if I'm mistaken, but what I see is that process_queue will grab the first value off the ready heap, and remove it. The removal process is to swap take the very last item in the heap, move it to the top, then siftdown and siftup. Both siftdown and siftup, however, only use priority for sorting, and if the priorities are all the same, then nothing will change. As such, the queue ordering becomes a little more random. As the ready heap shrinks, this is less of a problem, but if you're sitting on a heap that has hundreds or thousands of ready items, with more items being added regularly, the behavior becomes non-fifo as one might expect. I understand that beanstalk is intended to be used as a priority queue, and as such this functionality may not be a concern, but it doesn't seem to be documented as far as I could find, and it might be valuable information for others. It could make sense to some how use the age or original deadline_at as a secondary sort, but this isn't something that I've thought enough on to be confident of that sort of behavior. - Dustin -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/groups/opt_out.
