On Tue, Nov 3, 2009 at 7:51 PM, Dustin <[email protected]> wrote: > 1) Pagination. > > #1 will obviously give you a point in time.
Let's say you have a tube with 10M jobs, and we're giving 10 jobs per page. If you ask for page 500,000, we don't want the server to sift through 5M jobs in the binary heap just to find the 10 you care about. This would be slow and block other commands too. We could just grab them directly out of the underlying array, but then they'd come out in a weird order, and you are practically guaranteed to miss some as the jobs jump around whenever a new one is inserted or removed. A third option would be to use some other data structure that supports efficient range queries (in which case we might as well rip out the binary heap entirely). But this would cause a small performance hit all over beanstalkd and make performance less consistent. Hmm, here's a fourth way. Keep the jobs in a linked list along side the binary heap. Give each connection a cursor that just steps through the linked list. The order would not be the same as what you get from reserve, but at least it would make some sense. The only jobs you miss would have been removed from the queue entirely. This relies on connections being stateful. A new connection would always have to start paging at the first page -- you could never jump straight to the middle (or if you did it would be slow and block other things). I'm not particularly excited about any of these strategies. I'd love opinions on which is preferable. > My only concern is that #3 sounds like it'd be used for evil more than > good. Yeah, and I'm not sure it's really that necessary. You can always submit a new job and delete the old one. This gives you a different job id, but otherwise it's the same. kr --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en -~----------~----~----~----~------~----~------~--~---
