On Fri, Aug 13, 2010 at 2:20 PM, dustismo <[email protected]> wrote: > Hi, > > I am evaluating beanstalk for use in production environment. I have a > few questions: > > 1. throughput : we need to be able to handle ~ 100k tasks a minute
No problem. There are some performance number floating around out there. Here's one example: http://adam.heroku.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/ Of course, this all strongly depends on the hardware you use. Further, even if you do hit the limit of throughput on a single beanstalkd (which, to my knowledge, no one has ever done in production), it is trivial to distribute across two or more beanstalkd instances. > 2. Memory: Reading through the docs, it looks like new tasks will go > into a buried state if the queue does not fit in main memory. Is that > correct? So the task will become available again once the memory is > available? And if binary log is active the queue can grow as big as > space will allow? This isn't quite right. There are two separate issues here, out-of-memory conditions and the binlog. Although the doc says that beanstalkd can bury a job right when the client submits it, this essentially never happens in practice. When you submit a job, beanstalkd first mallocs space for the job struct. If this fails, the server will reply OUT_OF_MEMORY. Then, it puts the job in the array of jobs ready to be executed. If the array isn't big enough, beanstalkd will malloc space for a bigger array. If *this* malloc fails, then beanstalkd "buries" the job and replies BURIED. (Burying never needs to malloc because the buried jobs are kept in a linked list.) However, the success of malloc is usually not tied to the amount of physical memory in your machine, because of virtual memory, swap space, and ulimit settings. The binlog does not affect beanstalkd's capacity to store jobs. Regardless of whether you use the binlog, all jobs are always kept in memory. The binlog is only for disaster recovery. It is normally only written, never read. > 3. Ordering: We are interested in using as a LIFO queue rather then > FIFO. is this possible? Not easily. You can approximate it by giving your jobs a priority value based on the order of submission -- later jobs get a more urgent priority. But this approach has some serious drawbacks. kr -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
