Hi Collin, I happen to be in the same situation, ie seeing 'kernel serial8250 too much work for irq4' in /var/log/syslog, on Ubuntu 14.04, in Amazon EC2, with beanstalkd running. However, after following your solution, ie creating /etc/security/limits.d/beanstalkd.conf with the same content per your post and put 'ulimit -n 64000' in /etc/init.d/beanstalkd, I still see the problem. Please advise.
On Friday, October 3, 2014 5:05:43 AM UTC+7, Collin Hayden wrote: > > Hi all, > > I am currently running instances on the latest ubuntu 14.04.1 LTS image in > amazon ec2. I am having several issues that I have discovered and need > some help in confirming how the application currently works and functions. > > First, does the application run or hand off to root? I have noticed that > when using the package from ubuntu (1.9-2ubuntu1), that the installation > contains both the /etc/init.d/beanstalkd script and the > /etc/default/beanstalkd file to make optional adjustments to the > configuration. I have played around with persistence and not persistence > and it works fine. However I did notice that the logging in the init > script is not setup yet. I was able to set the logging to a file by > modifying the init script so that it redirects stdout and err to a file. I > noticed that the init script is set to run the application as the user > "beanstalkd". I set the ulimit for the beanstalkd user to 64000 for soft > and hard open files since we were hitting the limit of 1024 (the default in > linux). However, when I use the benchmark tool and open example 5000 > connections I noticed that beanstalkd is able to open the connections but > then in the log it hands off it root and root's user starts complaining > about too many open files. If I don't set my logging in the init script > and leave it as the default install, the root entries go to the console and > because there are thousands of connections it ends up over-flowing the > serial buffer for tty0 and tty1 ( I opened a ticket w/ amazon regarding > this and it seems they use the ttyS0 and ttyS1 for outputting console > messages to the dashboard for the instance in ec2). The issue here is that > the box gets hung up if you hit it w/ enough benchmark connections because > the ttyS0 gets overflowed and the kern.log will show messages like: > > "kernel: [ 311.699800] serial8250: too much work for irq4" > > The solution is to set the ulimit for the root user to 64000 in the > /etc/security/ulimit.conf file or /etc/security/limits.d/beanstalkd (i > created this file with entries so that it is read). Here is what mine > looks like: > > cat /etc/security/limits.d/beanstalkd.conf > > #<domain> <type> <item> <value># > > beanstalkd soft nofile 64000 > > beanstalkd hard nofile 64000 > > # Adding root as it appears that it is needed to function for beanstalkd > > root soft nofile 64000 > > root hard nofile 64000 > > > However from what I have read here: > http://linux.die.net/man/5/limits.conf and > http://linux.die.net/man/8/pam_limits , the limit.conf applies to pam. I > set the common-session and common-session-noninteractive to both honor the > pam_limits.so so that when you auth using pam in a shell that the ulimits > reflect appropriately. I gave beanstalkd a shell so I could test the > ulimit by doing a "sudo su - beanstalkd && ulimit -Sa" and see the correct > openfiles. This works fine. The ulimits are reflected for both root and > beanstalkd when using pam. I however noticed that upon reboot that during > the init process the beanstalkd daemon boots up but the beanstalkd doesn't > use pam apparently as the ulimits don't take effect and when benchmarking, > it crashes the machine because the output of the console messages is too > great for the ttyS0 (location where they are going w/o log redirection and > default setup) and hangs the box. So my main issue is how to set the > ulimit for the root user during initial boot? > > I know that you can set it in upstart, mongodb comes with this and here is > an example: http://docs.mongodb.org/manual/reference/ulimit/ for the > user that the script runs as. Right now it is not an upstart script, do > you guys have one? I have seen that the github project has a simple one, > but the users are complaining it doesn't work in this list. The main issue > is that even with an upstart script, you cannot set the ulimit for the root > user, since it appears that the ulimit is necessary for root in order to > open up enough sockets (open files). It will be set to whoever runs the > process and the init script, that initially is beanstalkd. > > The reason why this is a big issue is that we are currently looking to use > the persistence for our implementation and what I discovered is that you > need an equal amount of RAM for the amount of persistant data on disk in > the binlog files that are created in /var/lib/beanstalkd. When the > beanstalk comes up, if there is a huge amount of persistent messages it > takes time for it to get into memory so it appears the hang out for a > while, but in actuality it's loading the bin data into memory. We have > instances setup to have the sufficient amount of memory for the total > messages that need to persist for us. Right now we need just one day, so > 16 gig on an EC2 M3 XL instance is sufficient. I noticed that you cannot > free up the memory if using persistence by just blowing away the files, you > actually have to restart the beanstalkd in order for the process to free up > the memory. So for us we are planning to crontab something to make it so > that we reboot the instance once a day and delete the bin logs and free up > the memory. But if our system crashes during the day we are fine since we > have processed the messages we needed to. We are going to make the clients > "connection smart" so that they will re-connect to beanstalkd if the > process goes away. In order to implement this we want the beanstalkd to > setup the ulimits on boot up through the init process, how can we go about > this if root needs to be setup? The other question is why is root needed, > why can' t the app run purely under beanstalkd user? Right now I have to > restart the application in a shell and because the shell is using pam and > the process starts with the ulimit being acknowledged through pam, things > work. But this won't work for the cron restart to free up memory. I need > the ulimit to be honored during init for root. > > Suggestions on how I can accomplish this? > > Another question I have is: "Is the application running under Pam?" It > doesn't seem to be by what I have seen and implied above. > > I can share the modified init script that contains the log settings and > also the /etc/default/beanstalkd if others need an example of getting > verbose logging to work. A note here, is that when the logging is set the > crash in amazon doesn't happen as the messages are output to the file and > not the serial. I can also show examples of how I setup benchmarking. I > used the following: https://github.com/fangli/beanstalkd-benchmark > > The throughput on beanstalk is pretty amazing... thanks. > > > Best Regards, > > > Collin > -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/beanstalk-talk. For more options, visit https://groups.google.com/d/optout.
