Hi guys,
Apologies if this has been posted already but I couldn't find any
related topics in the archives. I saw what looked like a crash in my
couchdb server the other day - the relevant log section is included
below. I'm running couch version 1.2.0 on an Amazon EC2 instance running
Ubuntu 12.04.1.
It took about a day for this issue to get noticed and during that time
the server basically ran out of disk space:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 8.0G 7.6G 0 100% /
And creating small files is possible but creating large ones fails. It
seems possible that the disk filled around the time of the the couch
crash because the end of the log file does seem to be truncated (the
last thing I see is the "{summary, [" and "<<128,0,0,154>>," lines which
seem to have been cut off before the closing braces).
Also after the crash happened couch didn't seem to actually shutdown, as
I could still see the couchdb erlang process running and couch seemed to
be stuck in a state where no clients could connect to it. After I
stopped couch and cleared off the drive and then restarted the machine
clients could connect to it again and all seemed fine.
However, it's not obvious to me from the log / shutdown report that this
was related to disk space at all. Should I expect couch to log a warning
in a low disk space situation? Should this situation be recoverable? Can
anyone from the list help shed some light on what happened here?
Thanks,
Mark
[Mon, 15 Oct 2012 07:39:50 GMT] [info] [<0.22545.20>] 10.123.89.44 - -
GET /jungle/264529?conflicts=true 200
[Mon, 15 Oct 2012 07:39:50 GMT] [error] [<0.174.0>] ** Generic server
<0.174.0> terminating
** Last message in was {update_docs,<0.23330.20>,
[[{{doc,<<"2709-31ff">>,
{1,
[<<222,44,166,107,227,110,24,30,56,61,153,
116,242,241,242,189>>]},
{summary,
[<<128,0,0,154>>,
<<120,60,65,52,122,142,23,21,233,172,79,
5,217,11,12,237>>,
<<131,104,2,109,0,0,0,139,131,104,1,
108,0,0,0,4,104,2,109,0,0,0,9,116,
[Mon, 15 Oct 2012 07:39:51 GMT] [error]
[<0.23326.20>] Uncaught server error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23328.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22409.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22545.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22410.20>] Uncaught server
error: {kill,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23325.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.20484.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22547.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.22378.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23251.20>] Uncaught server
error: {noproc,
{gen_server,call,
[couch_server,
{open,<<"jungle">>,
[{user_ctx,
{user_ctx,null,
[<<"_admin">>],
<<"{couch_httpd_auth, default_authentication_handler}">>}}]},
infinity]}}
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23326.20>] 10.123.89.44 - -
GET /jungle/7452545?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23328.20>] 10.123.89.44 - -
GET /jungle/1883403?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22409.20>] 10.123.89.44 - -
GET /jungle/7508765?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22545.20>] 10.123.89.44 - -
GET /jungle/3713848?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22410.20>] 10.123.89.44 - -
GET /jungle/5153254?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23325.20>] 10.123.89.44 - -
GET /jungle/4742903?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.20484.20>] 10.123.89.44 - -
GET /jungle/5890907?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22547.20>] 10.123.89.44 - -
GET /jungle/9617720?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.22378.20>] 10.123.89.44 - -
GET /jungle/912831?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [info] [<0.23251.20>] 10.123.89.44 - -
GET /jungle/4130419?conflicts=true 500
[Mon, 15 Oct 2012 07:39:51 GMT] [error] [<0.23967.20>] ** Generic server
<0.23967.20> terminating
** Last message in was {update_docs,<0.23333.20>,
[[{{doc,<<"2709-31ff">>,
{1,
[<<222,44,166,107,227,110,24,30,56,61,153,
116,242,241,242,189>>]},
{summary,
[<<128,0,0,154>>,
