Thanks much for the responses. I'm going to try it out, and see how it goes. I just first need to write a pooled java driver (the current driver is GPL which won't work for us).
I think I understand the out of memory issues. Wondering if it is possible to set a finite amount of memory that app uses? What are the ramifications of going to swap? On rare occasions our processing nodes go down, so the queue can get backlogged. I'd rather get out of memory errors that I can handle client side then slowing down the queue server. plan for approximate lifo is what you suggested. (start at 3b and count down as tasks are submitted, then cycle when priority = 1). obviously its not perfect, but we don't really need it to be (just MOSTLY lifo). are there any other drawbacks to this approach? Thanks again for the help. On Fri, Aug 13, 2010 at 3:11 PM, Keith Rarick <[email protected]> wrote: > On Fri, Aug 13, 2010 at 2:20 PM, dustismo <[email protected]> wrote: >> Hi, >> >> I am evaluating beanstalk for use in production environment. I have a >> few questions: >> >> 1. throughput : we need to be able to handle ~ 100k tasks a minute > > No problem. There are some performance number floating around out > there. Here's one example: > > http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/ > > Of course, this all strongly depends on the hardware you use. > > Further, even if you do hit the limit of throughput on a single > beanstalkd (which, to my knowledge, no one has ever done in > production), it is trivial to distribute across two or more beanstalkd > instances. > >> 2. Memory: Reading through the docs, it looks like new tasks will go >> into a buried state if the queue does not fit in main memory. Is that >> correct? So the task will become available again once the memory is >> available? And if binary log is active the queue can grow as big as >> space will allow? > > This isn't quite right. There are two separate issues here, > out-of-memory conditions and the binlog. > > Although the doc says that beanstalkd can bury a job right when the > client submits it, this essentially never happens in practice. When > you submit a job, beanstalkd first mallocs space for the job struct. > If this fails, the server will reply OUT_OF_MEMORY. Then, it puts the > job in the array of jobs ready to be executed. If the array isn't big > enough, beanstalkd will malloc space for a bigger array. If *this* > malloc fails, then beanstalkd "buries" the job and replies BURIED. > (Burying never needs to malloc because the buried jobs are kept in a > linked list.) > > However, the success of malloc is usually not tied to the amount of > physical memory in your machine, because of virtual memory, swap > space, and ulimit settings. > > The binlog does not affect beanstalkd's capacity to store jobs. > Regardless of whether you use the binlog, all jobs are always kept in > memory. The binlog is only for disaster recovery. It is normally only > written, never read. > >> 3. Ordering: We are interested in using as a LIFO queue rather then >> FIFO. is this possible? > > Not easily. You can approximate it by giving your jobs a priority > value based on the order of submission -- later jobs get a more urgent > priority. But this approach has some serious drawbacks. > > kr > > -- > You received this message because you are subscribed to the Google Groups > "beanstalk-talk" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/beanstalk-talk?hl=en. > > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
