Hello again, I've had the time this weekend to implement some of the features I was looking for, my patches provide:
* the ability to retrieve a list of ids of all ready jobs in a tube * the ability to retrieve a list of ids of all reserved jobs in a tube (from all connections using the same tube) * the ability to retrieve a list of ids of all buried jobs in a tube This has given me the information I need, since once I got my hands on the ids, I could peek(id) at each Job I need to parse stats for. I will try to briefly explain what I've done and changed: I've had to add no extra data; all I needed was already being stored in the system, but there was no way to retrieve them through the protocol, which is a positive thing. - For the ready jobs, I used the static list all_jobs maintained in job.c that contains references to all ready jobs in all the tubes. I would iterate over the list, find the jobs that reside in the tube I'm using, store their id values in my dump array, then send it back to the client. - For the buried jobs, it was similar to the ready ones, except that I received a handler to the list from the Tube object itself. - For the reserved jobs, however, it proved to be a bit trickier; since reserved jobs were being stored within each Connection object exclusively. First, I retrieved the container of connections, I'd iterate through them to find the ones which are currently using the tube my connection is (these contain the reserved jobs that I'm after) and store references to each. I'd then parse each Connection's jobs within the `reserved_jobs' list, grab their id values and then finally dumping all the ids I processed into one big array which will be sent back to the client. There are a few things I could not accomplish due to the reserved job-connection exclusivity, though; I only get to peek a reserved job via a connection other than the one reserving it, but it's not possible to release, delete, or bury them. Attempting to get around that would've required me to change a major part of the flow within the system, and I thought it unnecessary. I'd probably break more things than introduce :p Throughout implementing these features, I was wondering if I'm supposed to do this at all. But for me, this allows me to monitor all the jobs I need; see their bodies, their stats, delete and bury them. It might not be very practical for instances with thousands of jobs to monitor, but that is not my case. In the light of this thread, I would also like to share my fork of the Ruby client for beanstalkd, Beanstalk-Client-Ruby; I modified their connection interface to reflect the new additions. You can pull the changes from: git://github.com/sugarfly/beanstalk-client-ruby.git Or alternatively, you can install the gem itself by: $ gem sources -a http://gems.github.com (you only have to do this once) $ sudo gem install sugarfly-beanstalk-client-ruby Below is also the simple Sinatra-based ruby web application I developed for the monitoring task, Beanstalker. You can feel free to use and modify it at your will. It enables you to browse all tubes, list ready, reserved, buried jobs within any from a specified beanstalkd server, with the ability to Delete ready jobs, and Kick buried ones. Found at: http://github.com/sugarfly/beanstalker Finally, you can pull my patches from below, they were merged with no conflicts after a recent clean pull from your repo (i hosted the repo on my server). Although I've done my best testing the changes under all possible scenarios I could simulate, I encourage you to review and test the changes, as I am no fluent C programmer. git://powerpuffcow.com/beanstalkd.git Thanks, Ahmad On Wed, Jan 20, 2010 at 2:34 AM, Keith Rarick <[email protected]> wrote: > On Tue, Jan 19, 2010 at 5:41 AM, Ahmad Amireh <[email protected]> > wrote: > > I'm attempting to create a web front-end for 'managing' our beanstalkd > > server, and by managing I mean the ability to: > > * browse watched and used tubes > > * browse jobs belonging to a certain tube {ready | buried | delayed} > > * bury a job, delete it, or modify its priority > > * view currently active / reserved jobs > > > > ... > > > > I'm really confused because it's looking to me like it's alot of > > trouble to retrieve this data or manage jobs. How do you guys do it? > > And am I way off with what i'm trying to accomplish? > > So far I haven't seen a lot of interest in doing these sorts of > things, but maybe that is just because the self-selected set of people > who post on this list or talk to me haven't been turned off by these > shortcomings. I can certainly understand the desire to have more > visibility into the queue and control over the jobs. > > I see no fundamental reason why the bury command shouldn't work on > ready jobs, or why you should not be able to change a ready job's > priority. > > However, I don't know how useful these things will be in a production > system, unless it's very lightly loaded. Beanstalkd is designed to > process many millions of jobs per day. At that rate, a tool that > reserves an entire tube to list out the jobs will become a bottleneck. > Similarly, you might not get the chance to bury or re-prioritize jobs > because they will fly by too fast. > > That said, I'd certainly accept patches to do these things. And I'm > still thinking about a way to iterate over the contents of a tube. > There was a discussion about that a while back, but so far I haven't > seen or thought of any good solutions. If anybody has further thoughts > on that, I'm all ears. > > kr > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<beanstalk-talk%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. > > > > -- There are 10 types of people, those who understand binary, and those who don't. -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
