Martin Sarsale wrote: > On Wed, Sep 16, 2009 at 1:13 AM, Yun Huang Yong <[email protected]> wrote: >> martin sarsale wrote: >>> We have it running since a few days (uptime: 1114539) and I just found >>> out that it's using a lot of memory even if the queue is almost empty: >>> >>> 8376 beanstalkd 20 0 1813m 1.7g 388 S 0 44.1 18:19.30 >>> beanstalkd >>> >>> this is the output of stats: >>> >>> current-jobs-ready: 2917 >>> current-jobs-reserved: 143 >>> current-jobs-buried: 8803 >>> max-job-size: 10000000 >> Hmmm these stats seem to indicate that your queues are not really >> "empty"? In total that's almost 12,000 jobs. How large are the jobs? > > jobs are tiny, definitely much smaller than 141666b/each (1.7g/12000). > We had to raise max-job-size because we were using in a different way before.
What OS are you on? Do your queue lengths spike much? If its Linux I wonder if you are bumping into memory fragmentation issues. In brief testing under Linux many months ago we found that in certain conditions beanstalkd's in-memory size would grow steadily over time if the queue sizes are allowed to spike. We think this is because of the way the pq/ms structures double in size when growth is needed. As they're allocated on the heap and semi-permanent you end up with an ever increasing high water mark over time. We're not sure if this problem would really manifest in real world usage since I haven't seen anyone mention it before but you can link against jemalloc to make it go away. http://www.canonware.com/~ttt/2008/08/stand-alone-jemalloc-for-linux.html Of course an even simpler explanation might be a memory leak somewhere... :) yun -- Yun Huang Yong [email protected] ...nom nom nom +61 408 131 419 -- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en -~----------~----~----~----~------~----~------~--~---
