Re: Workers reaped with SIGABRT - how to debug?
Henrik Nyh hen...@barsoom.se wrote: It was simply that monit did a kill -6 (SIBABRT) when the process used too much memory, so we bumped that limit for now. D'oh. We've yet to research why it used that much memory. Thanks for the followup. Unfortunately the RGenGC in Ruby 2.1.x uses more memory than 2.0 did (but GC is faster :). Sam wrote an article about it here: http://samsaffron.com/archive/2014/04/08/ruby-2-1-garbage-collection-ready-for-production We (ruby-core devs) will try to reduce memory for 2.2 without performance regressions. __ http://bogomips.org/unicorn-public/ - unicorn-pub...@bogomips.org please quote as little as necessary when replying
Re: Workers reaped with SIGABRT - how to debug?
Henrik, we had the same problem after upgrading to Ruby 2.1.1. My coworker Tieg Zaharia tracked it down to this bug with BigDecimal#to_d: https://bugs.ruby-lang.org/issues/9657 He discusses a workaround (using BigDecimal coercion instead of #to_d). Worked for us. Thank you! We'll look into that. We've definitely seen that error in CI. I couldn't find any segfaults in our production logs by the way, but I'm still not sure exactly where they would go. Have looked in our production.log (at the default prod log level) and the unicorn.log. (Sidenote: I think you have to CC everyone, not just reply to the list, or it only goes to the web archives.) ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Re: Workers reaped with SIGABRT - how to debug?
Henrik Nyh hen...@barsoom.se wrote: We get errors like this one a few times a day: Apr 13 12:16:31 app1 unicorn.log: E, [2014-04-13T12:16:31.302011 #17269] ERROR -- : reaped #Process::Status: pid 17300 SIGABRT (signal 6) worker=2 We use Unicorn 4.8.2, Ruby 2.1.1 and a Ruby on Rails app. It doesn't seem to happen at any obvious time, like during or just after deploys. We were previously on Ruby 1.9.3 with Unicorn 4.8.0. Then we had almost the same issue but with SIGIOT, I believe. Then we upgraded Ruby to 2.1.1. I believe that's when it changed to SIGABRT. Then we upgraded Unicorn to 4.8.2 with no improvement. We're not sure how to debug this - any suggestions on either what the problem could be, or how to debug it? This is may be a bug in a C extension RubyGem or even Ruby itself. Do you get core dumps + backtraces or any other error messages in the logs? Which OS/distribution is this? Since you see SIGABRT/SIGIOT and not SIGSEGV, you might be crashing inside the SIGSEGV handler of Ruby itself. Can you also try --enable-debug-env when you ./configure Ruby? Thanks in advance for any more info you can provide! (Btw, please keep everybody Cc:-ed since the mailing list is going in that direction, and rubyforge.org isn't very reliable in its final days. New ML announcement in a few days). ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Re: Workers reaped with SIGABRT - how to debug?
On Tue, Apr 15, 2014 at 10:43 AM, Eric Wong normalper...@yhbt.net wrote: This is may be a bug in a C extension RubyGem or even Ruby itself. Do you get core dumps + backtraces or any other error messages in the logs? You'd expect this to be in the Rails app's production.log, right? We added that to our logging service this morning - we'll check it the next time this error happens. Maybe we could also try to dig it up from archived logs. Which OS/distribution is this? Ubuntu 12.04.1 LTS Since you see SIGABRT/SIGIOT and not SIGSEGV, you might be crashing inside the SIGSEGV handler of Ruby itself. Can you also try --enable-debug-env when you ./configure Ruby? I guess that would output more details to the Rails production.log in the event of a crash? Will look into that. Thanks in advance for any more info you can provide! Thanks so much for the help! Much appreciated. ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
