Heya Greg, this should definitely not happen at all, regardless of AWS storage type.
Are there any other things going on on the VM, when you do this? Can you reliably reproduce this behaviour? Are there other correlating factors (like does this always happen at the same time / due to a cronjob, etc)? Can you set your CouchDB log level to debug and see if that gets you more info? (curl -X PUT http://[user:pass@]127.0.0.1:5984/_config/log/level -d '"debug"'). Is it possible for you to share these database files (publicly or in private)? What are your disk usage levels before/during compaction? Are you getting anything in the system log(s)? Best Jan -- Professional Support for Apache CouchDB: https://neighbourhood.ie/couchdb-support/ > On 07 Mar 2016, at 21:27, Greg Tarsa <[email protected]> wrote: > > We have a set of couchdb databases that we use to collect user information > for various purposes. I am inheriting this configuration from a predecessor > and am relatively new to couchdb. > > Whenever we attempt to compact the databases, the server crashes without any > messages either in the couchdb log or the system logs. This is running in an > AWS instance with an EBS volume. > > Experiments have shown that if the instance is configured with instance > storage (ephemeral storage that disappears when the instance disappears) then > this operation works properly. But we would like to use larger volumes and > have persistence. > > When the instance is configured with an external EBS volume, then we see the > server crash described above. > > I have searched the web for “couchdb compaction crash no log” and not found > anything helpful. > > It seems like compacting while running should not be failing at all, much > less silently, so I am looking for insights to the problem, or solutions if > such exist. > > Configuration and log info is below. > > Any help would be appreciated. > > Thanks, > Greg > > > --------------------------------------------------------- > > CouchDB version: 1.6.1 > OS: RHEL 6.6 > > --------------------------------------------------------- > > Here is a directory of the databases as the time of the crash: > > cat bad.couch.dbinfo.txt > total 15400740 > 12 -rw-r--r--. 1 couchdb couchdb 8297 Jan 20 16:31 _users.couch > 16 -rw-r--r--. 1 couchdb couchdb 12393 Jan 20 16:33 _replicator.couch > 21060 -rw-r--r--. 1 couchdb couchdb 21557368 Mar 7 11:57 biometrics.couch > 781136 -rw-r--r--. 1 couchdb couchdb 799875192 Mar 7 12:00 fitness.couch > 954244 -rw-r--r--. 1 couchdb couchdb 977137784 Mar 7 12:05 nutrition.couch > 8419624 -rw-r--r--. 1 couchdb couchdb 8621678721 Mar 7 12:06 routine.couch > 390796 -rw-r--r--. 1 couchdb couchdb 400167032 Mar 7 12:06 sleep.couch > 217932 -rw-r--r--. 1 couchdb couchdb 223154296 Mar 7 12:06 weight.couch > 4614884 -rw-r--r--. 1 couchdb couchdb 4725629060 Mar 7 12:06 trackers.couch > 4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41 > fitness.couch.compact > 4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41 > nutrition.couch.compact > 4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41 > routine.couch.compact > 64 -rw-r--r--. 1 couchdb couchdb 61551 Mar 7 12:41 diabetes.couch > 4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41 > sleep.couch.compact > 12 -rw-r--r--. 1 couchdb couchdb 8300 Mar 7 12:41 > tobacco_cessation.couch > 4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41 users.couch > 4 -rw-r--r--. 1 couchdb couchdb 79 Mar 7 12:41 > weight.couch.compact > 152 -rw-r--r--. 1 couchdb couchdb 151797 Mar 7 12:42 > trackers.couch.compact > 784 -rw-r--r--. 1 couchdb couchdb 801865 Mar 7 12:42 > biometrics.couch.compact > > --------------------------------------------------------- > > Here is the contents of the log at the time of the crash: > > [Mon, 07 Mar 2016 17:25:32 GMT] [info] [<0.31.0>] Apache CouchDB has started > on http://0.0.0.0:5984/ > [Mon, 07 Mar 2016 17:25:32 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:33 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:34 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:35 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:25:37 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:37 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:38 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:38 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:40 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:25:45 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:25:50 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:50 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:50 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:25:52 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:52 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:54 GMT] [info] [<0.114.0>] 10.1.1.12 - - GET > /users/_changes?feed=continuous&style=all_docs&since=0&heartbeat=10000 200 > [Mon, 07 Mar 2016 17:25:54 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:25:54 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:25:55 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:26:00 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:26:00 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /_active_tasks 200 > [Mon, 07 Mar 2016 17:26:01 GMT] [info] [<0.108.0>] 127.0.0.1 - - GET > /favicon.ico 200 > [Mon, 07 Mar 2016 17:26:05 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > ... [numerous GET /users/ 200 messages removed for brevity] ... > [Mon, 07 Mar 2016 17:41:51 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.152.0>] 127.0.0.1 - - GET > /_all_dbs 200 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1157.0>] Starting compaction for > db "biometrics" > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.151.0>] 127.0.0.1 - - POST > /biometrics/_compact 202 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.150.0>] 127.0.0.1 - - POST > /biometrics/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1175.0>] Starting compaction for > db "diabetes" > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.198.0>] 127.0.0.1 - - POST > /diabetes/_compact 202 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.197.0>] 127.0.0.1 - - POST > /diabetes/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1193.0>] Starting compaction for > db "fitness" > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.118.0>] 127.0.0.1 - - POST > /fitness/_compact 202 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.119.0>] 127.0.0.1 - - POST > /fitness/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.1211.0>] Starting compaction for > db "nutrition" > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.120.0>] 127.0.0.1 - - POST > /nutrition/_compact 202 > [Mon, 07 Mar 2016 17:41:52 GMT] [info] [<0.121.0>] 127.0.0.1 - - POST > /nutrition/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1229.0>] Starting compaction for > db "routine" > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.122.0>] 127.0.0.1 - - POST > /routine/_compact 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.115.0>] 127.0.0.1 - - POST > /routine/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1175.0>] Compaction for db > "diabetes" completed. > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1254.0>] Starting compaction for > db "sleep" > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.116.0>] 127.0.0.1 - - POST > /sleep/_compact 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.117.0>] 127.0.0.1 - - POST > /sleep/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1272.0>] Starting compaction for > db "tobacco_cessation" > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.184.0>] 127.0.0.1 - - POST > /tobacco_cessation/_compact 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.183.0>] 127.0.0.1 - - POST > /tobacco_cessation/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1290.0>] Starting compaction for > db "trackers" > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.182.0>] 127.0.0.1 - - POST > /trackers/_compact 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1272.0>] Compaction for db > "tobacco_cessation" completed. > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1151.0>] 127.0.0.1 - - POST > /trackers/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.162.0>] Starting compaction for db > "users" > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1152.0>] 127.0.0.1 - - POST > /users/_compact 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1168.0>] 127.0.0.1 - - POST > /users/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.162.0>] Compaction for db "users" > completed. > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1329.0>] Starting compaction for > db "weight" > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1170.0>] 127.0.0.1 - - POST > /weight/_compact 202 > [Mon, 07 Mar 2016 17:41:53 GMT] [info] [<0.1187.0>] 127.0.0.1 - - POST > /weight/_view_cleanup 202 > [Mon, 07 Mar 2016 17:41:56 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:42:01 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:42:06 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > [Mon, 07 Mar 2016 17:42:11 GMT] [info] [<0.123.0>] 10.1.1.12 - - GET /users/ > 200 > > -------------------------------------------------- >
