Morning Folks, As most of you know there were a few people who had the following three bugs:
* CLOSE_WAIT: Lots of sockets in CLOSE_WAIT state. * 99% CPU: Mongrel's getting "stuck" pegged at 99% CPU. * LEAK: Memory leak. I've successfully fixed these bugs or attributed them to one main cause: pstore. First, the memory leak was because of a bug in how the GC in Ruby was collecting threads after they used a Mutex. Don't ask me why, but switching to Sync fixed it. Problem is, this also causes the leak on Win32. Still working on that problem. Now for the recommendations: 1) DON'T USE PSTORE AS A SESSION STORE! All of the people who had Mongrel processes going crazy, blocking, locking, or just generally doing bad things had a pstore as their session storage option. What happens is the pstore isn't that great when multiple processes access it, and if one Mongrel has problems pulling an object from the pstore then it'll explode. Use the file store when you develop, and use the database or memcached store in production. You should also be looking for other ways processes are sharing resources when they lock up or do weird things. Try running with just one process, and if you don't have any problems then it's a resource locking problem (most likely). 2) DON'T STORE COMPLEX OBJECTS IN THE SESSION! This especially goes for you Java people who think the session store is a temporary database. The session really only works for simple types like integers, strings, etc. You *must* follow the mantra of "Share Nothing" and either store everything in the database or use form fields and GET parameters to make it stateless. As I've said repeatedly, "If rails-core don't do it, you don't do it." Rails-core does not do this. You don't do it. 3) USE THE USR1 LOGGING! Every person who had a problem with Mongrel processes getting stuck could have used the new USR1 logging to quickly find out what Rails action was blocking the process. Once I got a good USR1 log it was trivial to see a message stating that 981 threads were waiting for one Rails action. Once I pointed this out, they went back and figured out what the problem was. Use "killall -USR1 mongrel_rails" to turn this on temporarily, and Mongrel will log the numbers of threads waiting for each request and which are remaining to be processed by Mongrel. It's very handy. Hitting the mongrels with USR1 again turns it off. It's also pretty fast so you could leave it on all the time and not see a big performance hit. 4) YOUR RAILS CODE SUCKS! I hate to say it folks, and I'll gladly help people when they've got a problem, but please check your Rails code first, and assume it's broken. Lots and lots and lots and lots of sites are running high traffic Mongrel installs without any problems. If you have a problem, then you've got something different. I know, you're absolutely perfect and there's nothing you're doing strange, but really look. In all the cases mentioned above they were also doing something weird. Things to look for: * Using extensions to libraries you didn't write. If you have a library that you bought and then wrapped with Ruby's extensions, then chances are it's *not* designed for multi-threaded Ruby action. Move this library to a sequentially run DRb process shared by all your Mongrel processes. * Trying to resolve hosts against broken DNS. Many people have slow rails actions because they're doing tons of DNS resolving of hosts. Make sure your internal DNS is fast, reliable and consider using dnsmasq running locally to cache the results. Main problem is Ruby's resolving blocks the process. * Use pstore as the session store, so probably using pstore for nearly anything is bad. * Using your own custom network protocols. If you just wrote it then it's broken. #1, #2, and #3 rule of programming. It's not Mongrel. * Using the mysql.rb instead of installing the actual mysql compiled gem. The mysql.rb is just for development, and it sucks horribly. Thanks for your time folks, and keep coming with those bugs. -- Zed A. Shaw http://www.zedshaw.com/ http://mongrel.rubyforge.org/ http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help. _______________________________________________ Mongrel-users mailing list Mongrel-users@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users
