Hi,
I have a CouchDB instance v1.3.0 with about 6 databases and one of them is used
for historical reporting usually with 15 million records and about 30 GB to 45
GB disk space.
This Sunday it just crashed first rejecting http requests and then even killing
the couchDB process.
It is running in a Windows Server 2008 with about 32 GB of ram and 8 CPU cores.
While deleting a bunch of old records from the historical reports database this
error message pops-up
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.11190.28>] 15.128.50.71 - - DELETE
/outbound-record-history/6ebaaebd8f0bcffd18121a227a3654f0?rev=1-bf5e3f4fb9cc341b2c7be7599a4f7a8f
200
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.10928.28>] 15.128.50.71 - - DELETE
/outbound-record-history/6ebaaebd8f0bcffd18121a227a3651d8?rev=1-7aca019d84960522a12a8d2a7c224e90
200
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] 15.128.50.71 - - DELETE
/outbound-record-history/6ebaaebd8f0bcffd18121a227a3655d2?rev=1-467254a30672ad3c56d59ada50e81789
200
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Index shutdown by monitor
notice for db: outbound-record-history idx: _design/RecordHistory
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Index shutdown by monitor
notice for db: outbound-record-history idx: _design/FileFormatter
[Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13583.28>] Uncaught error in HTTP
request: {exit,
{{badmatch,
{error,eio}},
[{couch_btree,
'-write_node/3-lc$^0/1-0-',
5,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,358}]},
{couch_btree,
write_node,3,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,356}]},
{couch_btree,
modify_node,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,325}]},
{couch_btree,
modify_kpnode,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,394}]},
{couch_btree,
modify_node,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,315}]},
{couch_btree,
modify_kpnode,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,394}]},
{couch_btree,
modify_node,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,315}]},
{couch_btree,
modify_kpnode,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
{line,394}]}]}}
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Index shutdown by monitor
notice for db: orh-file-formatter idx: _design/default
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Index shutdown by monitor
notice for db: filters idx: _design/views
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Index shutdown by monitor
notice for db: _users idx: _design/_auth
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Index shutdown by monitor
notice for db: _replicator idx: _design/_replicator
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Closing index for db:
outbound-record-history idx: _design/RecordHistory sig:
"a2fb1653563e4d76e335ca6cd76f2730"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Closing index for db:
outbound-record-history idx: _design/FileFormatter sig:
"ab46065d7bb3b02cb49ebe6feee28445"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] Stacktrace:
[{couch_db,collect_results,3,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,828}]},
{couch_db,write_and_commit,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,840}]},
{couch_db,update_docs,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,777}]},
{couch_db,update_doc,4,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
{line,426}]},
{couch_httpd_db,update_doc,6,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
{line,753}]},
{couch_httpd_db,do_db_req,2,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
{line,234}]},
{couch_httpd,handle_request_int,5,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd.erl"},
{line,331}]},
{mochiweb_http,headers,5,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/mochiweb/mochiweb_http.erl"},
{line,136}]}]
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Closing index for db:
orh-file-formatter idx: _design/default sig: "332c4209ff6258e068dcb78d7c903ffc"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Closing index for db:
filters idx: _design/views sig: "dd3c6db3641c350bf7656b3b01e8ee1f"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Closing index for db: _users
idx: _design/_auth sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
[Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Closing index for db:
_replicator idx: _design/_replicator sig: "3e823c2a4383ac0c18d4e574135a5b08"
reason: normal
and then for each new request this error pops up
[Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13769.28>] Uncaught error in HTTP
request: {exit,
{{{badrecord,db},
[{couch_server,
handle_info,2,
[{file,
"c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_server.erl"},
{line,432}]},
{gen_server,
handle_msg,5,
[{file,
"gen_server.erl"},
{line,607}]},
{proc_lib,
init_p_do_apply,3,
[{file,
"proc_lib.erl"},
{line,227}]}]},
{gen_server,call,
[couch_server,
{open,
<<"outbound-record-history">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}}
The only way to restore the CouchDb was to delete the corrupted DB and restore
it from an earlier backup
Alejandro Santiago Femat
Senior Consultant - Technical Solution Group - North LATAM
Telefono +52.55.1101-6700 C +521.55.5505-1705
[email protected]<mailto:[email protected]>
www.genesyslab.com<http://www.genesyslab.com/>
