Jamie Wilkinson <ja...@tramchase.com> wrote: > I've also produced straces of the *original* master during USR2 > restarts, both a success trace and a failure trace. Here's a tarball > with both complete traces as well as filtered/grepp'd ones: > http://jamiedubs.com/files/unicorn-strace.tgz > > I've also found that kill -9'ing the 1st worker of the new orphaned > master allows it to continue operation as normal (spinning up workers > and taking control from the original master) -- suggesting something > is up with just that first worker (!). I'm going to keep noodling with > before_/after_fork strategies.
Hi Jamie, your successful straces have this oddity in them: poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0) = 1 ([{fd=7, revents=POLLIN|POLLHUP} read(7, ""..., 8192) = 0 shutdown(7, 2 /* send and receive */) = -1 ENOTCONN (Transport endpoint is not close(7) = 0 > Eric Wong <normalper...@yhbt.net> wrote: > > Anything in your before_fork/after_fork hooks? Since it looks like > > you're on a Linux system, can you strace the master while you send > > it a USR2 and see if anything strange happens? > > The only real contents of our before_hook is a > send-QUIT-on-first-worker, which I swapped out for the default SIGTTOU > behavior. No change. It looks like you should be disconnecting/reconnecting whatever is using fd=7 in the before_fork/after_fork hooks respectively. Unicorn (and mainline Ruby) do not use the poll() system call (which is why that strace raised red flags), so there's another C extension calling poll() in your master process (not good). I remember one (or all of) the Ruby Postgres libraries using poll() internally, and there are likely other things that use poll() in Ruby as well. Perhaps something like this in the config (assuming you're using Rails ActiveRecord) (also see http://unicorn.bogomips.org/examples/) before_fork do |server,worker| defined?(ActiveRecord::Base) and ActiveRecord::Base.connection.disconnect! end after_fork do |server,worker| defined?(ActiveRecord::Base) and ActiveRecord::Base.establish_connection end -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying