Hi, After years without a single problem, I think I might have seen my first issue with beanstalkd.
The context : We have a mecanism to control the flow of the messages. When we have "too much" messages in a queue from an ID or from a user, we put these messages in delayed state. We experienced an unusual amount of messages in a queue from an ID. So almost all the messages in this queue was set to delayed over an extensive period of time. We will patch this control mecanism but it showed interesting things about beanstalkd : - It generated a higher beanstalkd CPU usage (an expected behavior with more charge) - As the messages were almost always set to delayed, lots of network IO were generated between application servers and beanstalkd server. - After a while in this state (hours, days, it depends of the submitted load) : disk iowait reach 100% on the beanstalkd log disk, beanstalkd become unresponsive, the network IO stops immediatly. Most of the time beanstalkd recovered by itself after 20 or 30 minutes of freeze/iowait @ 100%. But sometimes we had to stop watching the queue and restart beanstalkd to speed up the recovery. The freeze/iowait period was very easy to predict. If the first freeze happened 20 hours after the submission of hundreds of thousands messages (which are flow controlled), the next one will happen in ~20 hours after recovery. At first I thought it was a machine, docker or linux related problem but I reproduced the problem quite a few times : - on a docker with Fedora 22 and beanstalkd-1.9-3.fc22.x86_64 over CoreOS, - on CentOS 6 and beanstalkd-1.9-2.el6.x86_64 on standalone machines (a medium sized AWS instance without SSD and a bigger one with SSD storage). Here's the pattern of the problem, always the same : Beanstalkd machine side : Queue side We didn't try with last beanstalkd version as the changelog didn't reveal anything about this kind of problem. Regards, Arthur -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout.
