> On 11 Dec 2014, at 00:53 , Alejandro Santiago
> <[email protected]> wrote:
>
> Hi,
>
> I have a CouchDB instance v1.3.0 with about 6 databases and one of them is
> used for historical reporting usually with 15 million records and about 30 GB
> to 45 GB disk space.
> This Sunday it just crashed first rejecting http requests and then even
> killing the couchDB process.
>
> It is running in a Windows Server 2008 with about 32 GB of ram and 8 CPU
> cores.
>
> While deleting a bunch of old records from the historical reports database
> this error message pops-up
>
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.11190.28>] 15.128.50.71 - - DELETE
> /outbound-record-history/6ebaaebd8f0bcffd18121a227a3654f0?rev=1-bf5e3f4fb9cc341b2c7be7599a4f7a8f
> 200
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.10928.28>] 15.128.50.71 - - DELETE
> /outbound-record-history/6ebaaebd8f0bcffd18121a227a3651d8?rev=1-7aca019d84960522a12a8d2a7c224e90
> 200
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] 15.128.50.71 - - DELETE
> /outbound-record-history/6ebaaebd8f0bcffd18121a227a3655d2?rev=1-467254a30672ad3c56d59ada50e81789
> 200
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Index shutdown by monitor
> notice for db: outbound-record-history idx: _design/RecordHistory
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Index shutdown by monitor
> notice for db: outbound-record-history idx: _design/FileFormatter
> [Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13583.28>] Uncaught error in HTTP
> request: {exit,
> {{badmatch,
> {error,eio}},
error,eio points to rather serious issues with your disks:
case ERROR_ARENA_TRASHED:
case ERROR_INVALID_BLOCK:
case ERROR_BAD_ENVIRONMENT:
case ERROR_BAD_COMMAND:
case ERROR_CRC:
case ERROR_OUT_OF_PAPER:
case ERROR_READ_FAULT:
case ERROR_WRITE_FAULT:
case ERROR_WRONG_DISK:
case ERROR_NET_WRITE_FAULT:
return EIO;
default: /* not to do with files I expect. */
return EIO;
https://github.com/erlang/otp/blob/maint/erts/emulator/drivers/win32/win_efile.c#L256-268
I don’t think CouchDB can do much in the face of disk errors like this (and out
of paper! :) bur crash.
Best
Jan
--
> [{couch_btree,
>
> '-write_node/3-lc$^0/1-0-',
> 5,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,358}]},
> {couch_btree,
> write_node,3,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,356}]},
> {couch_btree,
> modify_node,4,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,325}]},
> {couch_btree,
> modify_kpnode,6,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,394}]},
> {couch_btree,
> modify_node,4,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,315}]},
> {couch_btree,
> modify_kpnode,6,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,394}]},
> {couch_btree,
> modify_node,4,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,315}]},
> {couch_btree,
> modify_kpnode,6,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_btree.erl"},
> {line,394}]}]}}
>
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Index shutdown by monitor
> notice for db: orh-file-formatter idx: _design/default
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Index shutdown by monitor
> notice for db: filters idx: _design/views
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Index shutdown by monitor
> notice for db: _users idx: _design/_auth
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Index shutdown by monitor
> notice for db: _replicator idx: _design/_replicator
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.195.0>] Closing index for db:
> outbound-record-history idx: _design/RecordHistory sig:
> "a2fb1653563e4d76e335ca6cd76f2730"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.202.0>] Closing index for db:
> outbound-record-history idx: _design/FileFormatter sig:
> "ab46065d7bb3b02cb49ebe6feee28445"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.13583.28>] Stacktrace:
> [{couch_db,collect_results,3,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,828}]},
> {couch_db,write_and_commit,4,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,840}]},
> {couch_db,update_docs,4,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,777}]},
> {couch_db,update_doc,4,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_db.erl"},
> {line,426}]},
> {couch_httpd_db,update_doc,6,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
> {line,753}]},
> {couch_httpd_db,do_db_req,2,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd_db.erl"},
> {line,234}]},
> {couch_httpd,handle_request_int,5,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_httpd.erl"},
> {line,331}]},
> {mochiweb_http,headers,5,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/mochiweb/mochiweb_http.erl"},
> {line,136}]}]
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.218.0>] Closing index for db:
> orh-file-formatter idx: _design/default sig:
> "332c4209ff6258e068dcb78d7c903ffc"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.234.0>] Closing index for db:
> filters idx: _design/views sig: "dd3c6db3641c350bf7656b3b01e8ee1f"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.152.0>] Closing index for db:
> _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e574135a5b08"
> reason: normal
> [Sun, 07 Dec 2014 18:41:43 GMT] [info] [<0.159.0>] Closing index for db:
> _replicator idx: _design/_replicator sig: "3e823c2a4383ac0c18d4e574135a5b08"
> reason: normal
>
> and then for each new request this error pops up
>
> [Sun, 07 Dec 2014 18:41:43 GMT] [error] [<0.13769.28>] Uncaught error in HTTP
> request: {exit,
> {{{badrecord,db},
> [{couch_server,
> handle_info,2,
> [{file,
>
> "c:/cygwin/relax/APACHE~1.0/src/couchdb/couch_server.erl"},
> {line,432}]},
> {gen_server,
> handle_msg,5,
> [{file,
>
> "gen_server.erl"},
> {line,607}]},
> {proc_lib,
> init_p_do_apply,3,
> [{file,
> "proc_lib.erl"},
> {line,227}]}]},
> {gen_server,call,
> [couch_server,
> {open,
>
> <<"outbound-record-history">>,
> [{user_ctx,
> {user_ctx,null,
> [<<"_admin">>],
>
> <<"{couch_httpd_auth, default_authentication_handler}">>}}]},
> infinity]}}}
>
> The only way to restore the CouchDb was to delete the corrupted DB and
> restore it from an earlier backup
>
>
> Alejandro Santiago Femat
> Senior Consultant - Technical Solution Group - North LATAM
> Telefono +52.55.1101-6700 C +521.55.5505-1705
> [email protected]<mailto:[email protected]>
> www.genesyslab.com<http://www.genesyslab.com/>
