This is on a snapshot of the 0.10 branch and may be fixed already. Since
we're seeing this on a production server and not our development
environments, it's hard to write a failing test.
After a lot of successful (after a fashion[*]) PUTting to and DELETEing
from _config, we're seeing a sudden error in the logs, after which all
subsequent ones seem to fail. The first failure looks like this
(potential security sensitive info *****ed, but looks similar to earlier
successful attempts):
[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.13623.4>] 'PUT'
/_config/admins/50415 {1,1}
Headers: [{'Accept',"application/json"},
{'Accept-Encoding',"identity"},
{'Authorization',"OAuth realm=\"\", oauth_nonce=\"*****\",
oauth_timestamp=\"1253842863\", oauth_consumer_key=\"*****\",
oauth_signature_method=\"HMAC-SHA1\", oauth_version=\"1.0\",
oauth_token=\"*****\", oauth_signature=\"*****\""},
{'Content-Length',"10"},
{'Host',"raspberry.canonical.com:9030"},
{'User-Agent',"couchdb-python ?"}]
[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.13623.4>] OAuth Params:
[{"oauth_nonce","03532062"},
{"oauth_timestamp","1253842863"},
{"oauth_consumer_key","*****"},
{"oauth_signature_method","HMAC-SHA1"},
{"oauth_version","1.0"},
{"oauth_token","*****"},
{"oauth_signature","*****"}]
[Fri, 25 Sep 2009 01:41:03 GMT] [debug] [<0.44.0>] saving to file
'/etc/couchdb/local.ini', Config: '{{"admins","50415"},
"password"}'
[Fri, 25 Sep 2009 01:41:08 GMT] [error] [<0.13623.4>] Uncaught error in
HTTP request: {exit,
{timeout,
{gen_server,call,
[couch_config,
{set,"admins","50415","password",true}]}}}
[Fri, 25 Sep 2009 01:41:08 GMT] [info] [<0.13623.4>] Stacktrace:
[{gen_server,call,2},
{couch_httpd_misc_handlers,handle_config_req,1},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
[Fri, 25 Sep 2009 01:41:08 GMT] [debug] [<0.13623.4>] httpd 500 error
response:
{"error":"timeout","reason":"{gen_server,call,[couch_config,{set,\"admins\",\"50415\",\"password\",true}]}"}
[Fri, 25 Sep 2009 01:41:08 GMT] [info] [<0.13623.4>] 91.189.90.36 - -
'PUT' /_config/admins/50415 500
[Fri, 25 Sep 2009 01:41:08 GMT] [debug] [<0.44.0>] saving to file
'/etc/couchdb/local.ini', Config: '{{"admins","50415"},
"-hashed-*****"}'
NOTE: We get back a 500 (why, I don't know) and after that, the hashed
password is happily written to the file anyway?! (Note also: for
succesful writes, the order is the same, at least in the log file, which
seems wrong: it sends back a 200 OK and then attempts the write to the
.ini file...)
After this, all PUTS to _config seem to fail. Unfortunately it appears
that one of my colleagues in their haste to fix this problem, has
removed the local.ini file and started with a fresh one, so it is
impossible for me to determine what state it was in. (I have asked an
admin who is allowed to look in those files, but there seems to be
nothing strange there now, and the admin user that was being PUT does
not appear there.)
[*] Succesful in that we have to keep trying until we succeed, sometimes
several hundred times. I think this is because some erlang bits need to
be restarted after each write, like the auth_handler? Perhaps that is
what failed to come back up at all after above mishap...
I've been seeing errors like this which seem to suggest that, but I suck
at reading erlang/couchdb tracebacks:
[Fri, 25 Sep 2009 06:52:16 GMT] [error] [<0.3054.5>] {error_report,<0.24.0>,
{<0.3054.5>,crash_report,
[[{initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}},
{pid,<0.3054.5>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[couch_config,
{get,"httpd","authentication_handlers",undefined}]}},
[{gen_server,call,2},
{couch_httpd,handle_request,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]},
{messages,[]},
{links,[<0.63.0>,#Port<0.42195>]},
{dictionary,[{jsonp,undefined}]},
{trap_exit,false},
{status,running},
{heap_size,6765},
{stack_size,24},
{reductions,14822}],
[]]}}
--
- eric casteleijn
http://thisfred.blogspot.com
