Ah, we ran into this very issue :)
rack_timeout uses SystemTimer, and it turns out that the database call
isn't interrupted or aborted by the trapped signal. That's why the
timeout doesn't fire until the call returns:
> Benchmark.realtime { SystemTimer.timeout_after(5) {
AR::Base.connection.select_value("SELECT pg_sleep(10)") } rescue nil }
=> 10.0038049221039
We uncovered the statement_timeout setting in Postgres, which will
kill queries that exceed the setting. Doco here:
http://www.postgresql.org/docs/9.1/static/runtime-config-client.html#GUC-STATEMENT-TIMEOUT
> AR::Base.connection.execute("SET statement_timeout TO '4s'")
=> "4s"
> Benchmark.realtime { SystemTimer.timeout_after(5) {
AR::Base.connection.select_value("SELECT pg_sleep(10)") } rescue nil }
=> 4.00161099433899
With that in hand, we set statement_timeout to one second less than
the request timeout, and handled the exception that's raised by the
timed out query (an AR::StatementInvalid in ActiveRecord's case) in a
piece of middleware.
A few gotchas:
1. We set the statement_timeout in a Rails initializer
(config/initializers/something.rb), but we had to add some additional
chrome because we did not want the timeout enabled for background
jobs, rake tasks, or the development/test environments. Web requests
only.
2. AR::StatementInvalid isn't particular to timed-out queries, so we
had to do some substring matching on the exception's message in order
to filter it properly in middleware. Specifically, we were looking for
"canceling statement due to statement timeout" (when the
statement_timeout is exceeded and Pg kills it) and "Timeout::Error"
(when SystemTimer's SIGALRM is trapped during the database call).
3. If you're using Rails and the query times out in a view, then ugh,
the exception is swallowed into a generic ActionView::Template::Error.
The message, however, remains original.
Best of luck!
Dave
On Wed, Jan 4, 2012 at 9:02 PM, Peter van Hardenberg <[email protected]> wrote:
> Hi Sorens,
>
> sorry I didn't get back to you earlier. PostgreSQL only checks for
> connection interruption between statements -- this is an unfortunate design
> decision and perhaps something that could be better emphasized with the
> project as a serious production problem. (In fact, I may go start a thread
> on this issue right now!)
>
> When this occurs, you could use pg_cancel_backend(<query pid>) IF you were a
> superuser. Unfortunately, in order to provide a stable service we can't give
> that out. Because of that, we've worked with the project to upstream a fix,
> which should arrive with 9.2, and we're currently looking at backporting
> that patch to all new databases as soon as it stabilizes.
>
> Other than that, I'm afraid there's not much you can do currently except try
> not to issue single statements which cause big problems. Really sorry I
> don't have better news to bring you. It's an issue we're obviously tracking
> closely and working on a solution for but there isn't one available yet.
>
> Best,
> Peter
> Heroku Postgres
>
>
> On Fri, Dec 16, 2011 at 10:12 AM, sorens <[email protected]> wrote:
>>
>> Our web app is running on heroku with a dedicated postgres DB. We have
>> a long query that we're fixing(separate issue), but we noticed that
>> the rack-timeout did not cut the request off at 15 seconds as
>> expected. I posted a question about it on StackOverflow.
>>
>>
>> http://stackoverflow.com/questions/8512540/a-slow-postgres-query-on-heroku-is-not-being-interrupted-by-rack-timeout
>>
>> Does anyone have insight into how I can make rack-timeout cut off this
>> request? While we're tracking down the query issue, having the query
>> take out the service is causing problems.
>>
>> Thanks,
>> Steve
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Heroku" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/heroku?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Heroku" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/heroku?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
