Re: Fwd: Support for Soft Timeout in Unicorn
Pierre Baillet o...@fotonauts.com wrote: Hello Unicorns, I've manage to create a simple middleware that replaces the soft timeout feature. You can have a look at it at http://gist.github.com/431451 Note that some weird Ruby interpreter behavior breaks at least the first level of the generated stacktrace (it indicates the actual method where the raise happened but the wrong line number and file). You should be able to avoid spawning a new thread for every request, as that can get very expensive on some systems. Maybe a global hash guarded by a mutex that tells the worker Thread which thread to raise on. The worker thread could sleep until the next timeout registered. But then again, take care to only spawn new threads in workers (with preload_app=true), as threads never get carried across fork. And threads may leave mutexes and such in a bad state when they vanish in the child. -- Eric Wong ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Re: Fwd: Support for Soft Timeout in Unicorn
Hello Unicorns, I've manage to create a simple middleware that replaces the soft timeout feature. You can have a look at it at http://gist.github.com/431451 Note that some weird Ruby interpreter behavior breaks at least the first level of the generated stacktrace (it indicates the actual method where the raise happened but the wrong line number and file). Thanks for your assistance, Cheers, -- Pierre. On Thu, Jun 3, 2010 at 9:40 PM, Pierre Baillet o...@fotonauts.com wrote: Ohai, On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath ch...@ozmm.org wrote: On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong normalper...@yhbt.net wrote: Actually, internally, Unicorn only knows about Rack and wraps older CGI-based Rails using the Unicorn::App::OldRails application (via Unicorn::CGIWrapper). unicorn_rails basically wraps up the following config for you, but you could achieve the same effect using unicorn and an explicitly written config.ru: That's what we do at GitHub. We're running Rails 2.2.2 and have a custom config.ru, thanks to Unicorn: That's great, thanks for the Idea, I'll look into rack middleware then. I'll let you know if I manage to have something clean enough. Cheers, -- Pierre Baillet o...@fotonauts.com http://www.fotopedia.com/ -- Pierre Baillet o...@fotonauts.com http://www.fotopedia.com/ ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Fwd: Support for Soft Timeout in Unicorn
Hi, HTML attachments are wasteful and thus rejected from the mailing list. On the other hand, it actually helps to include the patch itself (inline) so it's readable without a (human) context switch :) - Forwarded message from Pierre Baillet o...@fotonauts.com - Subject: Fwd: Support for Soft Timeout in Unicorn From: Pierre Baillet o...@fotonauts.com To: unic...@bogomips.org Hi, Just tried to send that through the ml, but it seems something went wrong... Cheers, -- Pierre. -- Forwarded message -- From: mongrel-unicorn-ow...@rubyforge.org Date: Thu, Jun 3, 2010 at 2:40 PM Subject: Support for Soft Timeout in Unicorn To: o...@fotonauts.com The message's content type was not explicitly allowed -- Forwarded message -- From: Pierre Baillet o...@fotonauts.com To: mongrel-unicorn@rubyforge.org Date: Thu, 3 Jun 2010 14:16:01 +0200 Subject: Support for Soft Timeout in Unicorn Hi, We use Unicorn at fotopedia since yesterday in production. We switched from Passenger due to an issue in the way Passenger was handling some error in our main application. Things run very well on Unicorn. We have also modified Unicorn to handle a soft timeout for its workers. The Unicorn timeout was killing the workers without any chance for us to catch the Rails stack trace and identify the issue. I've forked and branched Unicorn from github and added a soft_timeout configuration value that is used for long running workers. The workers now handle SIGABRT and will raise an exception. This will crash the Rails application if it can be crashed and force the framework to dump the stack trace in the logs. Let me know if this might be useful for other people and, why not, integrate that in the main Unicorn code ! http://github.com/octplane/unicorn/tree/SOFT_TIMEOUT_SUPPORT Cheers, -- Pierre Baillet o...@fotonauts.com http://www.fotopedia.com/ -- Pierre Baillet o...@fotonauts.com http://www.fotopedia.com/ - End forwarded message - ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Re: Fwd: Support for Soft Timeout in Unicorn
Pierre Baillet wrote: We use Unicorn at fotopedia since yesterday in production. We switched from Passenger due to an issue in the way Passenger was handling some error in our main application. Things run very well on Unicorn. We have also modified Unicorn to handle a soft timeout for its workers. The Unicorn timeout was killing the workers without any chance for us to catch the Rails stack trace and identify the issue. I've forked and branched Unicorn from github and added a soft_timeout configuration value that is used for long running workers. The workers now handle SIGABRT and will raise an exception. This will crash the Rails application if it can be crashed and force the framework to dump the stack trace in the logs. Let me know if this might be useful for other people and, why not, integrate that in the main Unicorn code ! Hi Pierre, I'm thinking there's a better way to do this without involving the master process. The current timeout implementation[1] is really the last resort, point-of-no-return situations when the workers are completely stuck/blocked and cannot respond to other signals[2]. If the worker can respond to SIGABRT (especially going through the interpreter and raising an exception), then that means it could technically respond to Thread#raise, too... Unfortunately, the current core Ruby timeout implementation is extremely naive and inefficient. It shouldn't be hard to write a better timeout implementation that reuses a single timer Thread which can be rearmed with every request. This would be doable as middleware, too, and if done carefully, even safely reusable in multi-threaded web servers. This would be a good addition to rack-contrib, even. I might consider doing it myself if I had time. [1] - I'm not completely happy with needing the current timeout implementation in the first place. I will at least redo it at some point (after Unicorn 1.x) to gain some scalability/performance (and perhaps lose some portability). [2] - SIGKILL and SIGSTOP are special, userspace has no mechanism to block/catch/ignore those signals, so we rely on the master process to deliver them. -- Eric Wong ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Re: Fwd: Support for Soft Timeout in Unicorn
Eric, On Thu, Jun 3, 2010 at 8:22 PM, Eric Wong normalper...@yhbt.net wrote: This would be doable as middleware, too, and if done carefully, even safely reusable in multi-threaded web servers. This would be a good addition to rack-contrib, even. I might consider doing it myself if I had time. I also think that doing this as a Rack middleware is probably the right way to do that kind of worker management. However the application we use at fotopedia is still Rails 2.1 based (hence not Rack compatible AFAIK). This probably means we'll have keep on maintaining this slightly hacked version on our own then. Thank you for Unicorn ! :) -- Pierre Baillet o...@fotonauts.com http://www.fotopedia.com/ ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Re: Fwd: Support for Soft Timeout in Unicorn
Ohai, On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath ch...@ozmm.org wrote: On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong normalper...@yhbt.net wrote: Actually, internally, Unicorn only knows about Rack and wraps older CGI-based Rails using the Unicorn::App::OldRails application (via Unicorn::CGIWrapper). unicorn_rails basically wraps up the following config for you, but you could achieve the same effect using unicorn and an explicitly written config.ru: That's what we do at GitHub. We're running Rails 2.2.2 and have a custom config.ru, thanks to Unicorn: That's great, thanks for the Idea, I'll look into rack middleware then. I'll let you know if I manage to have something clean enough. Cheers, -- Pierre Baillet o...@fotonauts.com http://www.fotopedia.com/ ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
Re: Fwd: Support for Soft Timeout in Unicorn
On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong normalper...@yhbt.net wrote: Actually, internally, Unicorn only knows about Rack and wraps older CGI-based Rails using the Unicorn::App::OldRails application (via Unicorn::CGIWrapper). unicorn_rails basically wraps up the following config for you, but you could achieve the same effect using unicorn and an explicitly written config.ru: That's what we do at GitHub. We're running Rails 2.2.2 and have a custom config.ru, thanks to Unicorn: http://gist.github.com/424352 ___ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
