pankaj-dahiya opened a new issue #316: CouchDB is restarting URL: https://github.com/apache/openwhisk-devtools/issues/316 CouchDB is restarting at any point of time, with average load. Logs: ubuntu@ip-192-168-2-172:~$ docker logs --tail 100 -f openwhisk_db_1 [info] 2020-03-06T06:29:06.017872Z nonode@nohost <0.206.0> -------- Apache CouchDB has started on http://any:5986/ [info] 2020-03-06T06:29:06.018025Z nonode@nohost <0.9.0> -------- Application couch started on node nonode@nohost [info] 2020-03-06T06:29:06.018117Z nonode@nohost <0.9.0> -------- Application ets_lru started on node nonode@nohost [error] 2020-03-06T06:29:06.032765Z nonode@nohost <0.256.0> -------- Could not get design docs for <<"shards/e0000000-ffffffff/local_subjects.1583327006">> error:{badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [error] 2020-03-06T06:29:06.032768Z nonode@nohost emulator -------- Error in process <0.264.0> with exit value: {badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [notice] 2020-03-06T06:29:06.035915Z nonode@nohost <0.275.0> -------- rexi_server : started servers [notice] 2020-03-06T06:29:06.037558Z nonode@nohost <0.279.0> -------- rexi_buffer : started servers [info] 2020-03-06T06:29:06.037660Z nonode@nohost <0.9.0> -------- Application rexi started on node nonode@nohost [info] 2020-03-06T06:29:06.054675Z nonode@nohost <0.9.0> -------- Application mem3 started on node nonode@nohost [info] 2020-03-06T06:29:06.054794Z nonode@nohost <0.9.0> -------- Application fabric started on node nonode@nohost [info] 2020-03-06T06:29:06.066158Z nonode@nohost <0.9.0> -------- Application chttpd started on node nonode@nohost [info] 2020-03-06T06:29:06.073771Z nonode@nohost <0.9.0> -------- Application couch_index started on node nonode@nohost [info] 2020-03-06T06:29:06.073841Z nonode@nohost <0.9.0> -------- Application couch_mrview started on node nonode@nohost [info] 2020-03-06T06:29:06.073894Z nonode@nohost <0.9.0> -------- Application couch_plugins started on node nonode@nohost [notice] 2020-03-06T06:29:06.104516Z nonode@nohost <0.90.0> -------- config: [features] scheduler set to true for reason nil [info] 2020-03-06T06:29:06.120000Z nonode@nohost <0.9.0> -------- Application couch_replicator started on node nonode@nohost [info] 2020-03-06T06:29:06.125201Z nonode@nohost <0.9.0> -------- Application couch_peruser started on node nonode@nohost [info] 2020-03-06T06:29:06.132083Z nonode@nohost <0.9.0> -------- Application ddoc_cache started on node nonode@nohost [info] 2020-03-06T06:29:06.141197Z nonode@nohost <0.9.0> -------- Application global_changes started on node nonode@nohost [info] 2020-03-06T06:29:06.141222Z nonode@nohost <0.9.0> -------- Application jiffy started on node nonode@nohost [info] 2020-03-06T06:29:06.145054Z nonode@nohost <0.9.0> -------- Application mango started on node nonode@nohost [info] 2020-03-06T06:29:06.148923Z nonode@nohost <0.9.0> -------- Application setup started on node nonode@nohost [info] 2020-03-06T06:29:06.148962Z nonode@nohost <0.9.0> -------- Application snappy started on node nonode@nohost [info] 2020-03-06T06:29:06.832398Z nonode@nohost <0.513.0> -------- Starting index update for db: shards/80000000-9fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832447Z nonode@nohost <0.519.0> -------- Starting index update for db: shards/a0000000-bfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832468Z nonode@nohost <0.512.0> -------- Starting index update for db: shards/60000000-7fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832461Z nonode@nohost <0.516.0> -------- Starting index update for db: shards/e0000000-ffffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832541Z nonode@nohost <0.515.0> -------- Starting index update for db: shards/00000000-1fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832558Z nonode@nohost <0.514.0> -------- Starting index update for db: shards/c0000000-dfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832602Z nonode@nohost <0.517.0> -------- Starting index update for db: shards/40000000-5fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832643Z nonode@nohost <0.518.0> -------- Starting index update for db: shards/20000000-3fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [notice] 2020-03-06T06:29:07.087833Z nonode@nohost <0.313.0> 040e83f052 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 311 [notice] 2020-03-06T06:29:09.317652Z nonode@nohost <0.593.0> 16f6381196 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 129 [notice] 2020-03-06T06:29:11.085734Z nonode@nohost <0.357.0> -------- couch_replicator_clustering : cluster stable [notice] 2020-03-06T06:29:11.403482Z nonode@nohost <0.593.0> 351f4fa8d8 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 37 [notice] 2020-03-06T06:29:11.590263Z nonode@nohost <0.405.0> -------- Started replicator db changes listener <0.735.0> [notice] 2020-03-06T06:29:13.511453Z nonode@nohost <0.593.0> d67f188579 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 18 [notice] 2020-03-06T06:29:15.610703Z nonode@nohost <0.593.0> 5a85946595 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 22 [notice] 2020-03-06T06:29:18.004197Z nonode@nohost <0.593.0> 9abc4ebad2 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 17 [info] 2020-03-06T06:29:20.093836Z nonode@nohost <0.519.0> -------- Starting index update for db: shards/a0000000-bfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [notice] 2020-03-06T06:29:20.119636Z nonode@nohost <0.593.0> 618c92dcda db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 27 [notice] 2020-03-06T06:29:21.992352Z nonode@nohost <0.279.0> -------- rexi_buffer : cluster stable [notice] 2020-03-06T06:29:21.992456Z nonode@nohost <0.275.0> -------- rexi_server : cluster stable [notice] 2020-03-06T06:29:22.201283Z nonode@nohost <0.593.0> 67e91ba1d6 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 15 [notice] 2020-03-06T06:29:24.589630Z nonode@nohost <0.593.0> 1542d318b3 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 303 [os_mon] cpu supervisor port (cpu_sup): Erlang has closed [os_mon] memory supervisor port (memsup): Erlang has closed [info] 2020-03-06T06:29:27.886118Z nonode@nohost <0.9.0> -------- Application couch_log started on node nonode@nohost [info] 2020-03-06T06:29:27.889343Z nonode@nohost <0.9.0> -------- Application folsom started on node nonode@nohost [info] 2020-03-06T06:29:27.917527Z nonode@nohost <0.9.0> -------- Application couch_stats started on node nonode@nohost [info] 2020-03-06T06:29:27.917672Z nonode@nohost <0.9.0> -------- Application khash started on node nonode@nohost [info] 2020-03-06T06:29:27.922342Z nonode@nohost <0.9.0> -------- Application couch_event started on node nonode@nohost [info] 2020-03-06T06:29:27.922469Z nonode@nohost <0.9.0> -------- Application hyper started on node nonode@nohost [info] 2020-03-06T06:29:27.926485Z nonode@nohost <0.9.0> -------- Application ibrowse started on node nonode@nohost [info] 2020-03-06T06:29:27.930253Z nonode@nohost <0.9.0> -------- Application ioq started on node nonode@nohost [info] 2020-03-06T06:29:27.930381Z nonode@nohost <0.9.0> -------- Application mochiweb started on node nonode@nohost [info] 2020-03-06T06:29:27.936093Z nonode@nohost <0.206.0> -------- Apache CouchDB 2.3.1 is starting. [info] 2020-03-06T06:29:27.936155Z nonode@nohost <0.207.0> -------- Starting couch_sup [notice] 2020-03-06T06:29:27.942717Z nonode@nohost <0.90.0> -------- config: [features] pluggable-storage-engines set to true for reason nil [info] 2020-03-06T06:29:28.011774Z nonode@nohost <0.206.0> -------- Apache CouchDB has started. Time to relax. [info] 2020-03-06T06:29:28.011841Z nonode@nohost <0.206.0> -------- Apache CouchDB has started on http://any:5986/ [info] 2020-03-06T06:29:28.011999Z nonode@nohost <0.9.0> -------- Application couch started on node nonode@nohost [info] 2020-03-06T06:29:28.012088Z nonode@nohost <0.9.0> -------- Application ets_lru started on node nonode@nohost [error] 2020-03-06T06:29:28.028307Z nonode@nohost emulator -------- Error in process <0.264.0> with exit value: {badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [error] 2020-03-06T06:29:28.028329Z nonode@nohost <0.256.0> -------- Could not get design docs for <<"shards/e0000000-ffffffff/local_subjects.1583327006">> error:{badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [notice] 2020-03-06T06:29:28.030082Z nonode@nohost <0.275.0> -------- rexi_server : started servers [notice] 2020-03-06T06:29:28.031924Z nonode@nohost <0.279.0> -------- rexi_buffer : started servers [info] 2020-03-06T06:29:28.032017Z nonode@nohost <0.9.0> -------- Application rexi started on node nonode@nohost [info] 2020-03-06T06:29:28.048755Z nonode@nohost <0.9.0> -------- Application mem3 started on node nonode@nohost [info] 2020-03-06T06:29:28.048783Z nonode@nohost <0.9.0> -------- Application fabric started on node nonode@nohost [info] 2020-03-06T06:29:28.059825Z nonode@nohost <0.9.0> -------- Application chttpd started on node nonode@nohost [info] 2020-03-06T06:29:28.067676Z nonode@nohost <0.9.0> -------- Application couch_index started on node nonode@nohost [info] 2020-03-06T06:29:28.067742Z nonode@nohost <0.9.0> -------- Application couch_mrview started on node nonode@nohost [info] 2020-03-06T06:29:28.067762Z nonode@nohost <0.9.0> -------- Application couch_plugins started on node nonode@nohost [notice] 2020-03-06T06:29:28.098336Z nonode@nohost <0.90.0> -------- config: [features] scheduler set to true for reason nil [info] 2020-03-06T06:29:28.114139Z nonode@nohost <0.9.0> -------- Application couch_replicator started on node nonode@nohost [info] 2020-03-06T06:29:28.119777Z nonode@nohost <0.9.0> -------- Application couch_peruser started on node nonode@nohost [info] 2020-03-06T06:29:28.127034Z nonode@nohost <0.9.0> -------- Application ddoc_cache started on node nonode@nohost [info] 2020-03-06T06:29:28.136469Z nonode@nohost <0.9.0> -------- Application global_changes started on node nonode@nohost [info] 2020-03-06T06:29:28.136509Z nonode@nohost <0.9.0> -------- Application jiffy started on node nonode@nohost [info] 2020-03-06T06:29:28.140539Z nonode@nohost <0.9.0> -------- Application mango started on node nonode@nohost [info] 2020-03-06T06:29:28.144531Z nonode@nohost <0.9.0> -------- Application setup started on node nonode@nohost [info] 2020-03-06T06:29:28.144553Z nonode@nohost <0.9.0> -------- Application snappy started on node nonode@nohost [info] 2020-03-06T06:29:28.458348Z nonode@nohost <0.513.0> -------- Starting index update for db: shards/80000000-9fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458363Z nonode@nohost <0.515.0> -------- Starting index update for db: shards/20000000-3fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458363Z nonode@nohost <0.512.0> -------- Starting index update for db: shards/00000000-1fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458417Z nonode@nohost <0.516.0> -------- Starting index update for db: shards/60000000-7fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458430Z nonode@nohost <0.519.0> -------- Starting index update for db: shards/40000000-5fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458528Z nonode@nohost <0.514.0> -------- Starting index update for db: shards/e0000000-ffffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458544Z nonode@nohost <0.517.0> -------- Starting index update for db: shards/c0000000-dfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458578Z nonode@nohost <0.518.0> -------- Starting index update for db: shards/a0000000-bfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [notice] 2020-03-06T06:29:29.196043Z nonode@nohost <0.313.0> 0860ed2981 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 791 [notice] 2020-03-06T06:29:31.299075Z nonode@nohost <0.593.0> ea308a5967 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 28 [notice] 2020-03-06T06:29:33.080007Z nonode@nohost <0.357.0> -------- couch_replicator_clustering : cluster stable [notice] 2020-03-06T06:29:33.407720Z nonode@nohost <0.593.0> c1f76fdb37 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 16 [notice] 2020-03-06T06:29:33.910140Z nonode@nohost <0.405.0> -------- Started replicator db changes listener <0.735.0> [notice] 2020-03-06T06:29:35.512161Z nonode@nohost <0.593.0> 27d2ecef6f db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 25 [notice] 2020-03-06T06:29:37.613436Z nonode@nohost <0.593.0> c5cdf30014 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 16 [notice] 2020-03-06T06:29:39.793311Z nonode@nohost <0.593.0> a585d7298a db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 92 [notice] 2020-03-06T06:29:41.899248Z nonode@nohost <0.593.0> e8bd3e076e db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 58 [notice] 2020-03-06T06:29:43.031047Z nonode@nohost <0.275.0> -------- rexi_server : cluster stable [notice] 2020-03-06T06:29:43.031109Z nonode@nohost <0.279.0> -------- rexi_buffer : cluster stable [notice] 2020-03-06T06:29:44.005069Z nonode@nohost <0.593.0> c550ea3b9e db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 17 erl_child_setup closed Crash dump is being written to: erl_crash.dump...[os_mon] cpu supervisor port (cpu_sup): Erlang has closed [os_mon] memory supervisor port (memsup): Erlang has closed
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
