strange couch error under load

[Shaun Lindsay]

Hey all,

We've been seeing this strange error in the couch logs and we're not sure what 
to make of it.  It seems like once this error starts happening, it continues to 
occur and corresponds to a general unresponsiveness of that particular couch 
node.  My first thought was that it was running out of file descriptors, but 
that doesn't seem to be the case (it still happens with a very high ulimit).

In this error the database it's trying to open is 'profiles22', but the error 
doesn't seem tied to a particular database (we see it happening on all of our 
profiles shards).

We built this couch instance from trunk r713244, if that helps.

Anyone have any insight on what's going on?
[Fri, 27 Mar 2009 18:21:42 GMT] [error] [<0.14547.0>] {error_report,<0.21.0>,
    {<0.14547.0>,crash_report,
     [[{pid,<0.14547.0>},
       {registered_name,[]},
       {error_info,
           {exit,
               {timeout,
                   {gen_server,call,
                       [couch_server,{open,<<"profiles22">>,[{creds,{[]}}]}]}},
               [{gen_server,call,2},
                {couch_httpd_db,do_db_req,2},
                {couch_httpd,handle_request,3},
                {mochiweb_http,headers,4},
                {proc_lib,init_p,5}]}},
       {initial_call,
           {mochiweb_socket_server,acceptor_loop,
               [{<0.59.0>,#Port<0.149>,#Fun<mochiweb_http.1.62821608>}]}},
       {ancestors,[couch_httpd,<0.53.0>,couch_server_sup,<0.1.0>]},
       {messages,[]},
       {links,[<0.59.0>,#Port<0.14390>]},
       {dictionary,[]},
       {trap_exit,false},
       {status,running},
       {heap_size,2584},
       {stack_size,23},
       {reductions,3300}],
      []]}}

Thanks,
Shaun