I am doing multimaster replication where each server is configured to push its
changes to the other one. The log I sent was from the remote server trying to
send the changes to my local server. As an experiment I instructed the local
server to pull from the remote and that is not working either. This is a
segment of the log in the local server:
[error] 2019-04-22T13:39:06.972000Z couchdb@localhost <0.20502.2490> --------
CRASH REPORT Process (<0.20502.2490>) with 1 neighbors exited with reason: no
match of right hand value {error,eio} at
couch_bt_engine_stream:write/2(line:60) <= couch_stream:do_write/2(line:302) <=
couch_stream:handle_call/3(line:278) <= gen_server:try_handle_call/4(line:615)
<= gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247) at
gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_stream,init,['Argument__1']}, ancestors: [<0.27672.2480>],
messages: [], links: [<0.27672.2480>], dictionary:
[{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.15...">>}}],
trap_exit: false, status: running, heap_size: 987, stack_size: 27, reductions:
596
[error] 2019-04-22T13:39:07.081000Z couchdb@localhost <0.18062.2491> --------
Replicator, request PUT to "http://
nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad07bc57-1:results?new_edits=false"
failed due to error {code,500}
[error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.4458.2476> --------
Replicator, request PUT to "http://
nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad0c5e1c-3:results?new_edits=false"
failed due to error {code,500}
[error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490> --------
gen_server <0.22292.2490> terminated with reason: no match of right hand value
{error,eio} at couch_bt_engine_stream:write/2(line:60) <=
couch_stream:do_write/2(line:302) <= couch_stream:handle_call/3(line:278) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647) <=
proc_lib:init_p_do_apply/3(line:247)
last msg: close
state:
{stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.74448900.31762>,[],[<<31,
(...etc...),0>>],2186,4096,0,<<>>,<<>>,2186,#Fun<couch_stream.8.80129967>,#Fun<couch_stream.9.80129967>}
[error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490> --------
CRASH REPORT Process (<0.22292.2490>) with 1 neighbors exited with reason: no
match of right hand value {error,eio} at
couch_bt_engine_stream:write/2(line:60) <= couch_stream:do_write/2(line:302) <=
couch_stream:handle_call/3(line:278) <= gen_server:try_handle_call/4(line:615)
<= gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247) at
gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_stream,init,['Argument__1']}, ancestors: [<0.16555.2491>],
messages: [], links: [<0.16555.2491>], dictionary:
[{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.15...">>}}],
trap_exit: false, status: running, heap_size: 987, stack_size: 27, reductions:
676
[error] 2019-04-22T13:39:07.627000Z couchdb@localhost <0.13178.2487> --------
Replicator, request PUT to "http://
nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad043479-71:results?new_edits=false"
failed due to error {code,500}
[error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483> --------
gen_server <0.30019.2483> terminated with reason: no match of right hand value
{error,eio} at couch_bt_engine_stream:write/2(line:60) <=
couch_stream:do_write/2(line:302) <= couch_stream:handle_call/3(line:278) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647) <=
proc_lib:init_p_do_apply/3(line:247)
last msg: close
state:
{stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.88604680.223648>,[],[<<31,(...etc...),0>>],2039,4096,0,<<>>,<<>>,2039,#Fun<couch_stream.8.80129967>,#Fun<couch_stream.9.80129967>}
[error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483> --------
CRASH REPORT Process (<0.30019.2483>) with 1 neighbors exited with reason: no
match of right hand value {error,eio} at
couch_bt_engine_stream:write/2(line:60) <= couch_stream:do_write/2(line:302) <=
couch_stream:handle_call/3(line:278) <= gen_server:try_handle_call/4(line:615)
<= gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247) at
gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_stream,init,['Argument__1']}, ancestors: [<0.4711.2491>],
messages: [], links: [<0.4711.2491>], dictionary:
[{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.151...">>}}],
trap_exit: false, status: running, heap_size: 987, stack_size: 27, reductions:
676
I have now restarted the service and replication briefly resumed but it is now
failing again. I'll continue to investigate.
Regards,
Gustavo Delfino
-----Original Message-----
From: Peter J Krawetzky <pkrawet...@gmail.com>
Sent: Sunday, April 21, 2019 3:08 PM
To: user@couchdb.apache.org
Subject: Re: CouchDB replication crashing
Are the errors below on the primary or the replication server? PeterSent from
my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: Delfino Gustavo
<gustavo.delf...@zf.com> Date: 4/21/19 13:25 (GMT-05:00) To:
user@couchdb.apache.org Subject: CouchDB replication crashing Hello, I have had
two CouchDB servers replicating a couple of DBs for several months. One server
is local and the other is in remote location. Both are running CouchDB 2.2.0
under Windows 7. Replication is failing and I know I can reactivate it by
restarting the remote server but I would like to understand if there is
anything I can do to prevent this situation in the first place.In previous
CouchDB versions, I used to have this problem where the replication seems to be
disconnected from the network while the DB continues to be accessible. It had
been working very well under 2.2.0. I wonder if moving up to 2.3 would help.A
representative segment of the log file is found below.Regards,Gustavo
Delfino[error] 2019-04-21T12:38:21.805000Z couchdb@localhost <0.28098.851>
-------- Replicator, request PUT to
"http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
failed due to error {code,500}[error] 2019-04-21T12:38:24.020000Z
couchdb@localhost <0.8055.852> -------- Replicator, request PUT to
"http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
failed due to error {code,500}[error] 2019-04-21T12:38:28.497000Z
couchdb@localhost <0.12289.852> -------- Replicator, request PUT to
"http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
failed due to error {code,500}[error] 2019-04-21T12:38:51.819000Z
couchdb@localhost <0.11004.852> -------- Replicator, request GET to
"http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-211?revs=true&open_revs=%5B%221-4c79aa0880ec04131b40c06855b36ff4%22%5D&latest=true";
failed due to error {error,req_timedout}[error] 2019-04-21T12:38:51.819000Z
couchdb@localhost <0.9866.852> -------- Replicator, request GET to
"http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-15?atts_since=%5B%222-597b62c56ad573a21a63e62c1117856e%22%5D&revs=true&open_revs=%5B%223-5f29e1678a58784ee2e8aee46c57b29d%22%5D&latest=true";
failed due to error sel_conn_closed[error] 2019-04-21T12:38:52.240000Z
couchdb@localhost <0.21284.852> -------- Replicator, request PUT to
"http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
failed due to error {code,500}[error] 2019-04-21T12:39:01.678000Z
couchdb@localhost <0.19310.852> -------- Replicator, request POST to
"http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to error
{code,500}[error] 2019-04-21T12:39:01.678000Z couchdb@localhost <0.9042.852>
-------- gen_server <0.9042.852> terminated with reason:
{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
last msg:
{'EXIT',<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
state:
[{main_pid,<0.9322.852>},{loop,<0.6467.852>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_fetch,nil},{batch_size,268141}][error]
2019-04-21T12:39:01.678000Z couchdb@localhost <0.9042.852> -------- CRASH
REPORT Process (<0.9042.852>) with 1 neighbors exited with reason:
{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
messages: [], links: [<0.9322.852>,<0.6467.852>], dictionary:
[{last_stats_report,{1555,850299,901000}}], trap_exit: true, status: running,
heap_size: 6772, stack_size: 27, reductions: 393291[error]
2019-04-21T12:39:18.916000Z couchdb@localhost <0.9687.852> -------- Replicator,
request POST to "http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed
due to error {code,500}[error] 2019-04-21T12:39:18.916000Z couchdb@localhost
<0.12947.851> -------- gen_server <0.12947.851> terminated with reason:
{process_died,<0.9687.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
last msg:
{'EXIT',<0.9687.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
state:
[{main_pid,<0.9322.852>},{loop,<0.15756.851>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_fetch,nil},{batch_size,383624}][error]
2019-04-21T12:39:18.916000Z couchdb@localhost <0.12947.851> -------- CRASH
REPORT Process (<0.12947.851>) with 1 neighbors exited with reason:
{process_died,<0.9687.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
messages: [], links: [<0.9322.852>,<0.15756.851>], dictionary:
[{last_stats_report,{1555,850299,839000}}], trap_exit: true, status: running,
heap_size: 6772, stack_size: 27, reductions: 552772[error]
2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> --------
Replicator, request POST to
"http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to error
{code,500}[error] 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852>
-------- gen_server <0.10607.852> terminated with reason:
{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}
at couch_replicator_httpc:report_error/4(line:363) <=
couch_replicator_httpc:send_req/3(line:76) <=
couch_replicator_worker:flush_docs/2(line:493) <=
couch_replicator_worker:maybe_flush_docs/3(line:459) <=
couch_replicator_worker:maybe_flush_docs/2(line:436) <=
couch_replicator_worker:handle_call/3(line:124) <=
gen_server:try_handle_call/4(line:615) <= gen_server:handle_msg/5(line:647)
last msg:
{batch_doc,{doc,<<"4bb0aa8269a879fdc95107973872dcfb-424">>,{3,[<<157,204,36,86,54,61,219,128,174,147,218,239,156,55,31,23>>,<<61,194,111,234,124,76,158,204,215,47,185,48,158,105,104,215>>,<<203,140,2,100,128,7,203,236,171,131,32,29,132,106,170,51>>]},
(...erlang json removed...) ,[],false,[]}} state:
[{main_pid,<0.9322.852>},{loop,<0.1165.852>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,20},{pending_fetch,{{<0.1165.852>,#Ref<0.0.23068677.99464>},{<<"4bb0aa8269a879fdc95107973872dcfb-174">>,[{3,<<139,102,22,196,97,193,111,166,119,37,58,206,57,73,182,130>>}],[{2,<<132,199,224,202,205,103,3,105,55,29,255,196,200,63,170,152>>}]}}},{batch_size,523788}][error]
2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> -------- CRASH
REPORT Process (<0.10607.852>) with 15 neighbors exited with reason:
{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}
at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
messages:
[{'EXIT',<0.24052.852>,normal},{'$gen_call',{<0.20083.852>,...},...},...],
links: [<0.22451.852>,<0.24660.852>,<0.25927.852>,<0.27245.852>,...],
dictionary:
[{last_stats_report,{1555,850301,711000}},{ibrowse_stream_status,...}],
trap_exit: true, status: running, heap_size: 28690, stack_size: 27, reductions:
744275[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852>
-------- Worker <0.9042.852> died with reason:
{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}[error]
2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> --------
Replication `6ccccb02a7483deae5c1c434bce9b376+continuous+create_target`
(`http://farawayhost.com:5984/jobhandlersims/` ->
`http://nearbyhost.com:5984/jobhandlersims/`) failed:
{worker_died,<0.9042.852>, {process_died,<0.19310.852>,
{http_request_failed,"POST",
"http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",
{error,{code,500}}}}}[error] 2019-04-21T12:39:27.995000Z couchdb@localhost
<0.9322.852> -------- gen_server
{couch_replicator_scheduler_job,{[54,99,99,99,99,98,48,50,97,55,52,56,51,100,101,97,101,53,99,49,99,52,51,52,98,99,101,57,98,51,55,54],[43,99,111,110,116,105,110,117,111,117,115,43,99,114,101,97,116,101,95,116,97,114,103,101,116]}}
terminated with reason:
{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
last msg:
{'EXIT',<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
state:
[{rep_id,{"6ccccb02a7483deae5c1c434bce9b376","+continuous+create_target"}},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{db_name,<<"shards/40000000-5fffffff/_replicator.1526394302">>},{doc_id,<<"DE_to_US_for_jobhandlersims">>},{options,[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{create_target,true},{http_connections,20},{retries,5},{selector,{[{<<"$or">>,[{[{<<"type">>,{[{<<"$ne">>,<<"results">>}]}}]},{[{<<"views">>,{[{<<"$gte">>,<<>>}]}}]},{[{<<"type">>,<<"results">>},{<<"requester_location">>,<<"US">>}]}]}]}},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}]},{session_id,<<"436547a63366ea84a38d8aeb36dd2d88">>},{start_seq,{0,<<"3194311-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsZqDDTMwsUizMDIg1bAEkGH1cMNUDCGGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQD5VMZk">>}},{source_seq,<<"3229426-g1AAAAFreJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8riYHtnzQedUkKQDLJHqqUTQ6fUgeQ0nioUokz-JQmgJTWQ5WaLcSjNI8FSDI0ACmg6vkg5R0PCSpfAFG-H6icNUKdoPIDEOX3QconXyCo_AFEOdjtdSVZAOiudpk">>},{committed_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{current_through_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{highest_seq_done,{8,<<"3204835-g1AAAAI3eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge3T91ygGLu5qWFqimEyNj14TEpSAJJJ9iDDkhhYf17Cp9QBpDQebi-3MNheAxOLNAsjA1LtTQAZVg-xl01zKR6leSxAkqEBSAFVz4fY3egIttsy0Sgx2dKCRLshBi6AGLgfbCCrVxPYQKO0REsLM2OyDDwAMfA-xMB2C0ismBslpyaakmXgA4iB0OAuSgcbaJKUkmxkmYhNaxYAmZez-A">>}}][error]
2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> -------- CRASH
REPORT Process (<0.9322.852>) with 3 neighbors exited with reason:
{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors:
[couch_replicator_scheduler_sup,couch_replicator_sup,...], messages:
[{'EXIT',<0.12947.851>,{process_died,<0.9687.852>,...}},...], links:
[<0.162.852>,<0.5422.852>,<0.10686.852>,<0.25534.850>,...], dictionary:
[{pending_count_state,{{1555,850367,995000},9348}},{task_status_update,...},...],
trap_exit: true, status: running, heap_size: 10958, stack_size: 27,
reductions: 63152[error] 2019-04-21T12:39:27.995000Z couchdb@localhost
<0.10686.852> -------- gen_server <0.10686.852> terminated with reason:
{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
last msg:
{'EXIT',<0.9322.852>,{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}}
state:
[{main_pid,<0.9322.852>},{loop,<0.6805.852>},{source,"http://farawayhost.com:5984/jobhandlersims/"},{target,"http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,1},{pending_fetch,nil},{batch_size,351185}][error]
2019-04-21T12:39:27.995000Z couchdb@localhost <0.10686.852> -------- CRASH
REPORT Process (<0.10686.852>) with 2 neighbors exited with reason:
{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
at gen_server:terminate/7(line:812) <= proc_lib:init_p_do_apply/3(line:247);
initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors:
[<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
messages: [], links: [<0.6805.852>,<0.20253.851>], dictionary:
[{last_stats_report,{1555,850299,839000}}], trap_exit: true, status: running,
heap_size: 2586, stack_size: 27, reductions: 496765[error]
2019-04-21T12:39:27.995000Z couchdb@localhost <0.385.0> -------- Supervisor
couch_replicator_scheduler_sup had child undefined started with
{couch_replicator_scheduler_job,start_link,undefined} at <0.9322.852> exit with
reason
{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
in context child_terminated[error] 2019-04-21T12:40:13.313000Z
couchdb@localhost <0.23845.851> 7ec5e9ec3e rexi_server: from:
couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:40:13.423000Z couchdb@localhost <0.19545.851> 7ec5e9ec3e
rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:40:13.438000Z couchdb@localhost <0.26362.851> 7ec5e9ec3e
rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:40:13.438000Z couchdb@localhost <0.25940.851> 7ec5e9ec3e
rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:40:13.438000Z couchdb@localhost <0.25878.851> 7ec5e9ec3e
rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:40:13.438000Z couchdb@localhost <0.25863.851> 7ec5e9ec3e
rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:40:13.438000Z couchdb@localhost <0.28372.851> 7ec5e9ec3e
rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:40:13.438000Z couchdb@localhost <0.29759.851> 7ec5e9ec3e
rexi_server: from: couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:43:23.323000Z couchdb@localhost <0.16895.850> ed368f9947
rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
2019-04-21T12:43:23.339000Z couchdb@localhost <0.3976.852> ed368f9947
rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]}][error]
2019-04-21T12:43:23.339000Z couchdb@localhost <0.3667.852> ed368f9947
rexi_server: from: couchdb@localhost(<0.29169.851>) mfa: fabric_rpc:changes/3
exit:timeout
[{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]}]
