Re: Unicorn_rails ignores USR2 signal

[RyanonRails]

We've run into the exact same problem. We spent a few good solid hours trying to
fix the problem with no avail (but we do have a 'solution').

Server specs:
Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-27-generic x86_64)

Rails  Ruby:
Rails 4.0.0.beta1
Ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux]
Unicorn 4.6.2 - preload_app true
Capistrano

The first thing we tried was removing all of the gems (foreman, sidekiq, etc.)
and that didn't work. We tried preload_app false, and that worked (but this is
not the situation we want). We also tried searching for any gems that were
capturing USR2 (or any signals in general).

We tried monkey patching Kernel  Trap to see if we could pinpoint the problem,
but that didn't seem to turn up much. We then jumped in unicorn's http_server.rb
and threw in some logging. The only thing that seemed to show us was that USR2
was unable to get through to unicorn. Signals HUP, QUIT, etc. all worked fine
(and made their way through as per norm).

We then fired up strace with unicorn to see if we could 'see' the USR2 signal.
It looks almost identical to Jeffery's strace. It looks like the USR2 signal is
received by the process, but then dropped (or ignored). If we pass it a QUIT
signal it works just fine and we can see that in the strace.

So here's the chain of events that we think is occuring:

1. We start unicorn (a unicorn blessing)
2. before_fork is triggered (Signal trapping works here)
3. A unicorn is born (Signal trapping does not work here)
4. The after fork is triggered

In our situation. Once a unicorn is born, it can no longer received a USR2 (for
whatever reason). We never actually checked to see if the after fork received a
USR2.

After we managed to figure out the path of execution, we figured out a quick
fix. We actually call reexec method directly on the unicorn server in the
before_fork block in unicorn.rb. Here's our unicorn.rb
https://gist.github.com/RyanonRails/5541902

- snip -

before_fork do |server, worker|
  Signal.trap 'USR2' do
puts 'Since a black hole is lurking and eating USR2s we will hit
http_server#reexec ourselves'
server.send(:reexec)
  end
end
- snip -

This way we can actually force the USR2 directly on the server (since USR2 is
available inside of the before_fork).

This seems to be working well for us now. We're still frustrated that we weren't
able to find the cause of the problem, but we least we can move forward (with
minor disruption to the code/code base).

Lastly, here are our current theories as to why this could be occuring:
1. When unicorn is building it's native extensions something weird is happening
in regards to the USR2 signal
2. A gem is somehow gobbling up the signal (we were quite thorough, but we
might've missed something)

Hopefully this helps!

Thanks,
Ryan  Mike


___
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

[PATCH] HttpParser#next? becomes response_start_sent-aware

[Eric Wong]

This could allow servers with persistent connection support[1]
to support our check_client_connection in the future.

[1] - Rainbows!/zbatery, possibly others
---
 ext/unicorn_http/unicorn_http.rl |  6 ++
 test/unit/test_http_parser_ng.rb | 17 +
 2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl
index 1a8003f..3529740 100644
--- a/ext/unicorn_http/unicorn_http.rl
+++ b/ext/unicorn_http/unicorn_http.rl
@@ -732,10 +732,8 @@ static VALUE HttpParser_parse(VALUE self)
   struct http_parser *hp = data_get(self);
   VALUE data = hp-buf;
 
-  if (HP_FL_TEST(hp, TO_CLEAR)) {
-http_parser_init(hp);
-rb_funcall(hp-env, id_clear, 0);
-  }
+  if (HP_FL_TEST(hp, TO_CLEAR))
+HttpParser_clear(self);
 
   http_parser_execute(hp, RSTRING_PTR(data), RSTRING_LEN(data));
   if (hp-offset  MAX_HEADER_LEN)
diff --git a/test/unit/test_http_parser_ng.rb b/test/unit/test_http_parser_ng.rb
index 93c44bb..ab335ac 100644
--- a/test/unit/test_http_parser_ng.rb
+++ b/test/unit/test_http_parser_ng.rb
@@ -12,6 +12,23 @@ def setup
 @parser = HttpParser.new
   end
 
+  def test_next_clear
+r = GET / HTTP/1.1\r\nHost: example.com\r\n\r\n
+@parser.buf  r
+@parser.parse
+@parser.response_start_sent = true
+assert @parser.keepalive?
+assert @parser.next?
+assert @parser.response_start_sent
+
+# persistent client makes another request:
+@parser.buf  r
+@parser.parse
+assert @parser.keepalive?
+assert @parser.next?
+assert_equal false, @parser.response_start_sent
+  end
+
   def test_keepalive_requests_default_constant
 assert_kind_of Integer, HttpParser::KEEPALIVE_REQUESTS_DEFAULT
 assert HttpParser::KEEPALIVE_REQUESTS_DEFAULT = 0
-- 
Eric Wong
___
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying