New user accounts cannot sign-in after _user database crashes
--------------------------------------------------------------
Key: COUCHDB-1212
URL: https://issues.apache.org/jira/browse/COUCHDB-1212
Project: CouchDB
Issue Type: Bug
Components: Database Core, HTTP Interface
Affects Versions: 1.0.2
Environment: Ubuntu 10.10, Erlang R14B02 (erts-5.8.3)
Reporter: Jan van den Berg
Priority: Critical
We have one (4,5 GB) database next to the default _users database. Every other
day we need to restart couchdb because newly sign-up user accounts cannot
login. They get a 401 from the _session HTTP interface. This is the stacktrace
I see in the couch database log prior to these problems.
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.84.0>] ** Generic server
couch_server terminating
** Last message in was {open,<<"fbm">>,
[{user_ctx,{user_ctx,null,[],undefined}}]}
** When Server state == {server,"/opt/couchbase-server/var/lib/couchdb",
{re_pattern,0,0,
<<69,82,67,80,116,0,0,0,16,0,0,0,1,0,0,0,0,0,
0,0,0,0,0,0,40,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,93,0,72,25,77,0,0,0,0,0,0,0,0,0,0,0,0,254,
255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
77,0,0,0,0,16,171,255,3,0,0,0,128,254,255,
255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,69,26,
84,0,72,0>>},
100,2,"Sat, 18 Jun 2011 14:00:44 GMT"}
** Reason for termination ==
** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.84.0>] {error_report,<0.31.0>,
{<0.84.0>,crash_report,
[[{initial_call,{couch_server,init,['Argument__1']}},
{pid,<0.84.0>},
{registered_name,couch_server},
{error_info,
{exit,
{timeout,
{gen_server,call,
[<0.116.0>,{open_ref_count,<0.10417.79>}]}},
[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
{ancestors,[couch_primary_services,couch_server_sup,<0.32.0>]},
{messages,[]},
{links,[<0.91.0>,<0.483.0>,<0.116.0>,<0.79.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,6765},
{stack_size,24},
{reductions,206710598}],
[]]}}
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.79.0>] {error_report,<0.31.0>,
{<0.79.0>,supervisor_report,
[{supervisor,{local,couch_primary_services}},
{errorContext,child_terminated},
{reason,
{timeout,
{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}},
{offender,
[{pid,<0.84.0>},
{name,couch_server},
{mfargs,{couch_server,sup_start_link,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]}]}}
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.91.0>] ** Generic server <0.91.0>
terminating
** Last message in was {'EXIT',<0.84.0>,
{timeout,
{gen_server,call,
[<0.116.0>,
{open_ref_count,<0.10417.79>}]}}}
** When Server state == {db,<0.91.0>,<0.92.0>,nil,<<"1308405644393791">>,
<0.90.0>,<0.94.0>,
{db_header,5,91,0,
{378285,{30,9}},
{380466,39},
nil,0,nil,nil,1000},
91,
{btree,<0.90.0>,
{378285,{30,9}},
#Fun<couch_db_updater.7.10053969>,
#Fun<couch_db_updater.8.35220795>,
#Fun<couch_btree.5.124754102>,
#Fun<couch_db_updater.9.107593676>},
{btree,<0.90.0>,
{380466,39},
#Fun<couch_db_updater.10.30996817>,
#Fun<couch_db_updater.11.96515267>,
#Fun<couch_btree.5.124754102>,
#Fun<couch_db_updater.12.117826253>},
{btree,<0.90.0>,nil,#Fun<couch_btree.0.83553141>,
#Fun<couch_btree.1.30790806>,
#Fun<couch_btree.2.124754102>,nil},
91,<<"_users">>,
"/opt/couchbase-server/var/lib/couchdb/_users.couch",
[#Fun<couch_doc.7.50754398>],
[],nil,
{user_ctx,null,[],undefined},
nil,1000,
[before_header,after_header,on_file_open],
true}
** Reason for termination ==
** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.91.0>] {error_report,<0.31.0>,
{<0.91.0>,crash_report,
[[{initial_call,{couch_db,init,['Argument__1']}},
{pid,<0.91.0>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[<0.116.0>,{open_ref_count,<0.10417.79>}]}},
[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.89.0>]},
{messages,[]},
{links,[]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,610},
{stack_size,24},
{reductions,8797798}],
[]]}}
[Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.300.0>] Shutting down view group
server, monitored db is closing.
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.10417.79>] Uncaught error in HTTP
request: {exit,
{{timeout,
{gen_server,call,
[<0.116.0>,
{open_ref_count,<0.10417.79>}]}},
{gen_server,call,
[couch_server,
{open,<<"fbm">>,
[{user_ctx,
{user_ctx,null,[],undefined}}]},
infinity]}}}
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.483.0>] ** Generic server <0.483.0>
terminating
** Last message in was {'EXIT',<0.84.0>,
{timeout,
{gen_server,call,
[<0.116.0>,
{open_ref_count,<0.10417.79>}]}}}
** When Server state == {db,<0.483.0>,<0.484.0>,nil,<<"1308405937993370">>,
<0.4643.19>,<0.4645.19>,
{db_header,5,890453,0,
{3279126950,{752003,0}},
{3279118313,752003},
{3279132318,[]},
0,nil,3279127184,1000},
890453,
{btree,<0.4643.19>,
{3279126950,{752003,0}},
#Fun<couch_db_updater.7.10053969>,
#Fun<couch_db_updater.8.35220795>,
#Fun<couch_btree.5.124754102>,
#Fun<couch_db_updater.9.107593676>},
{btree,<0.4643.19>,
{3279118313,752003},
#Fun<couch_db_updater.10.30996817>,
#Fun<couch_db_updater.11.96515267>,
#Fun<couch_btree.5.124754102>,
#Fun<couch_db_updater.12.117826253>},
{btree,<0.4643.19>,
{3279132318,[]},
#Fun<couch_btree.0.83553141>,
#Fun<couch_btree.1.30790806>,
#Fun<couch_btree.2.124754102>,nil},
890453,<<"fbm_full">>,
"/opt/couchbase-server/var/lib/couchdb/fbm_full.couch",
[#Fun<couch_doc.7.50754398>],
[{<<"admins">>,
{[{<<"names">>,[]},
{<<"roles">>,[<<"import">>]}]}},
{<<"readers">>,
{[{<<"names">>,[]},{<<"roles">>,[]}]}}],
3279127184,
{user_ctx,null,[],undefined},
nil,1000,
[before_header,after_header,on_file_open],
false}
** Reason for termination ==
** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}
[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.483.0>] {error_report,<0.31.0>,
{<0.483.0>,crash_report,
[[{initial_call,{couch_db,init,['Argument__1']}},
{pid,<0.483.0>},
{registered_name,[]},
{error_info,
{exit,
{timeout,
{gen_server,call,
[<0.116.0>,{open_ref_count,<0.10417.79>}]}},
[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.480.0>]},
{messages,[]},
{links,[]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,6765},
{stack_size,24},
{reductions,1389}],
[]]}}
[Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.2984.19>] Shutting down view group
server, monitored db is closing.
[Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.10417.79>] Stacktrace:
[{gen_server,call,3},
{couch_server,open,2},
{couch_db,open,2},
{couch_httpd_db,do_db_req,2},
{couch_httpd,handle_request_int,5},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
