Thank you Nick and Peter for your help. It is now back to normal.
I had plenty of disk space on CouchDB database hard disks, but the startup disk
was 99.5% full. Besides dealing with this space situation I ran the defragment
tool as I know that it can also be an issue under windows. Thankfully I'll be
moving this DB to Linux soon.
Regards,
Gustavo Delfino
-----Original Message-----
From: Nick Vatamaniuc <vatam...@gmail.com>
Sent: Monday, April 22, 2019 12:19 PM
To: user@couchdb.apache.org
Subject: Re: CouchDB replication crashing
Hi Gustavo,
"no match of right hand value {error,eio}" looks like a failure to write to a
block device. Perhaps disks are full, or there is some kind of throttling
(over-quota) issue?
Replication will restart on failures. There is an exponential backoff on
repeated errors so if your replications crash too many times in a row they'd be
penalized and wait a bit. But I'd investigate why there are these EIO errors
happening first.
Cheers,
-Nick
On Mon, Apr 22, 2019 at 11:39 AM Delfino Gustavo <gustavo.delf...@zf.com>
wrote:
> I am doing multimaster replication where each server is configured to
> push its changes to the other one. The log I sent was from the remote
> server trying to send the changes to my local server. As an experiment
> I instructed the local server to pull from the remote and that is not
> working either. This is a segment of the log in the local server:
>
> [error] 2019-04-22T13:39:06.972000Z couchdb@localhost <0.20502.2490>
> -------- CRASH REPORT Process (<0.20502.2490>) with 1 neighbors
> exited with reason: no match of right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <=
> couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <=
> proc_lib:init_p_do_apply/3(line:247)
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_stream,init,['Argument__1']}, ancestors: [<0.27672.2480>], messages:
> [], links: [<0.27672.2480>], dictionary:
> [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.
> 15...">>}}],
> trap_exit: false, status: running, heap_size: 987, stack_size: 27,
> reductions: 596
> [error] 2019-04-22T13:39:07.081000Z couchdb@localhost <0.18062.2491>
> -------- Replicator, request PUT to "http://
> nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad07bc57-1:results?new_edits=false"
> failed due to error {code,500}
> [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.4458.2476>
> -------- Replicator, request PUT to "http://
> nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad0c5e1c-3:results?new_edits=false"
> failed due to error {code,500}
> [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490>
> -------- gen_server <0.22292.2490> terminated with reason: no match of
> right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <=
> couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247)
> last msg: close
> state:
> {stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.74448900
> .31762>,[],[<<31,
> (...etc...),0>>],2186,4096,0,<<>>,<<>>,2186,#Fun<couch_stream.8.801299
> 67>,#Fun<couch_stream.9.80129967>}
> [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490>
> -------- CRASH REPORT Process (<0.22292.2490>) with 1 neighbors
> exited with reason: no match of right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <=
> couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <=
> proc_lib:init_p_do_apply/3(line:247)
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_stream,init,['Argument__1']}, ancestors: [<0.16555.2491>], messages:
> [], links: [<0.16555.2491>], dictionary:
> [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.
> 15...">>}}],
> trap_exit: false, status: running, heap_size: 987, stack_size: 27,
> reductions: 676
> [error] 2019-04-22T13:39:07.627000Z couchdb@localhost <0.13178.2487>
> -------- Replicator, request PUT to "http://
> nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad043479-71:results?new_edits=false"
> failed due to error {code,500}
> [error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483>
> -------- gen_server <0.30019.2483> terminated with reason: no match of
> right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <=
> couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247)
> last msg: close
> state:
> {stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.88604680
> .223648>,[],[<<31,(...etc...),0>>],2039,4096,0,<<>>,<<>>,2039,#Fun<cou
> ch_stream.8.80129967>,#Fun<couch_stream.9.80129967>}
> [error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483>
> -------- CRASH REPORT Process (<0.30019.2483>) with 1 neighbors
> exited with reason: no match of right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <=
> couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <=
> proc_lib:init_p_do_apply/3(line:247)
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_stream,init,['Argument__1']}, ancestors: [<0.4711.2491>], messages:
> [], links: [<0.4711.2491>], dictionary:
> [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.
> 151...">>}}],
> trap_exit: false, status: running, heap_size: 987, stack_size: 27,
> reductions: 676
>
> I have now restarted the service and replication briefly resumed but
> it is now failing again. I'll continue to investigate.
>
> Regards,
>
> Gustavo Delfino
>
>
> -----Original Message-----
> From: Peter J Krawetzky <pkrawet...@gmail.com>
> Sent: Sunday, April 21, 2019 3:08 PM
> To: user@couchdb.apache.org
> Subject: Re: CouchDB replication crashing
>
> Are the errors below on the primary or the replication server?
> PeterSent from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: Delfino Gustavo <
> gustavo.delf...@zf.com> Date: 4/21/19 13:25 (GMT-05:00) To:
> user@couchdb.apache.org Subject: CouchDB replication crashing Hello, I
> have had two CouchDB servers replicating a couple of DBs for several
> months. One server is local and the other is in remote location. Both
> are running CouchDB 2.2.0 under Windows 7. Replication is failing and
> I know I can reactivate it by restarting the remote server but I would
> like to understand if there is anything I can do to prevent this
> situation in the first place.In previous CouchDB versions, I used to
> have this problem where the replication seems to be disconnected from
> the network while the DB continues to be accessible. It had been
> working very well under 2.2.0. I wonder if moving up to 2.3 would
> help.A representative segment of the log file is found
> below.Regards,Gustavo Delfino[error] 2019-04-21T12:38:21.805000Z
> couchdb@localhost <0.28098.851> -------- Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:38:24.020000Z
> couchdb@localhost <0.8055.852> -------- Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:38:28.497000Z
> couchdb@localhost <0.12289.852> -------- Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:38:51.819000Z
> couchdb@localhost <0.11004.852> -------- Replicator, request GET to "
> http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-211?revs=true&open_revs=%5B%221-4c79aa0880ec04131b40c06855b36ff4%22%5D&latest=true";
> failed due to error {error,req_timedout}[error]
> 2019-04-21T12:38:51.819000Z couchdb@localhost <0.9866.852> --------
> Replicator, request GET to "
> http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-15?atts_since=%5B%222-597b62c56ad573a21a63e62c1117856e%22%5D&revs=true&open_revs=%5B%223-5f29e1678a58784ee2e8aee46c57b29d%22%5D&latest=true";
> failed due to error sel_conn_closed[error] 2019-04-21T12:38:52.240000Z
> couchdb@localhost <0.21284.852> -------- Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:39:01.678000Z
> couchdb@localhost <0.19310.852> -------- Replicator, request POST to "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to
> error {code,500}[error] 2019-04-21T12:39:01.678000Z couchdb@localhost
> <0.9042.852> -------- gen_server <0.9042.852> terminated with reason:
> {process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}} last msg: {'EXIT',<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}
> state: [{main_pid,<0.9322.852>},{loop,<0.6467.852>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_
> fetch,nil},{batch_size,268141}][error]
> 2019-04-21T12:39:01.678000Z couchdb@localhost <0.9042.852> --------
> CRASH REPORT Process (<0.9042.852>) with 1 neighbors exited with reason:
> {process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}} at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...]
> ,
> messages: [], links: [<0.9322.852>,<0.6467.852>], dictionary:
> [{last_stats_report,{1555,850299,901000}}], trap_exit: true, status:
> running, heap_size: 6772, stack_size: 27, reductions: 393291[error]
> 2019-04-21T12:39:18.916000Z couchdb@localhost <0.9687.852> --------
> Replicator, request POST to "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to
> error {code,500}[error] 2019-04-21T12:39:18.916000Z couchdb@localhost
> <0.12947.851> -------- gen_server <0.12947.851> terminated with reason:
> {process_died,<0.9687.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}} last msg: {'EXIT',<0.9687.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}
> state: [{main_pid,<0.9322.852>},{loop,<0.15756.851>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_
> fetch,nil},{batch_size,383624}][error]
> 2019-04-21T12:39:18.916000Z couchdb@localhost <0.12947.851> --------
> CRASH REPORT Process (<0.12947.851>) with 1 neighbors exited with reason:
> {process_died,<0.9687.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}} at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...]
> ,
> messages: [], links: [<0.9322.852>,<0.15756.851>], dictionary:
> [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status:
> running, heap_size: 6772, stack_size: 27, reductions: 552772[error]
> 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> --------
> Replicator, request POST to "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to
> error {code,500}[error] 2019-04-21T12:39:27.964000Z couchdb@localhost
> <0.10607.852> -------- gen_server <0.10607.852> terminated with reason:
> {http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}} at couch_replicator_httpc:report_error/4(line:363) <=
> couch_replicator_httpc:send_req/3(line:76) <=
> couch_replicator_worker:flush_docs/2(line:493) <=
> couch_replicator_worker:maybe_flush_docs/3(line:459) <=
> couch_replicator_worker:maybe_flush_docs/2(line:436) <=
> couch_replicator_worker:handle_call/3(line:124) <=
> gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) last msg:
> {batch_doc,{doc,<<"4bb0aa8269a879fdc95107973872dcfb-424">>,{3,[<<157,204,36,86,54,61,219,128,174,147,218,239,156,55,31,23>>,<<61,194,111,234,124,76,158,204,215,47,185,48,158,105,104,215>>,<<203,140,2,100,128,7,203,236,171,131,32,29,132,106,170,51>>]},
> (...erlang json removed...) ,[],false,[]}} state:
> [{main_pid,<0.9322.852>},{loop,<0.1165.852>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,20},{pending
> _fetch,{{<0.1165.852>,#Ref<0.0.23068677.99464>},{<<"4bb0aa8269a879fdc9
> 5107973872dcfb-174">>,[{3,<<139,102,22,196,97,193,111,166,119,37,58,20
> 6,57,73,182,130>>}],[{2,<<132,199,224,202,205,103,3,105,55,29,255,196,
> 200,63,170,152>>}]}}},{batch_size,523788}][error]
> 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> --------
> CRASH REPORT Process (<0.10607.852>) with 15 neighbors exited with reason:
> {http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}} at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...]
> ,
> messages:
> [{'EXIT',<0.24052.852>,normal},{'$gen_call',{<0.20083.852>,...},...},.
> ..],
> links: [<0.22451.852>,<0.24660.852>,<0.25927.852>,<0.27245.852>,...],
> dictionary:
> [{last_stats_report,{1555,850301,711000}},{ibrowse_stream_status,...}]
> ,
> trap_exit: true, status: running, heap_size: 28690, stack_size: 27,
> reductions: 744275[error] 2019-04-21T12:39:27.995000Z
> couchdb@localhost <0.9322.852> -------- Worker <0.9042.852> died with reason:
> {process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}[error] 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852>
> -------- Replication
> `6ccccb02a7483deae5c1c434bce9b376+continuous+create_target` (`
> http://farawayhost.com:5984/jobhandlersims/`
> <http://farawayhost.com:5984/jobhandlersims/> -> `
> http://nearbyhost.com:5984/jobhandlersims/`
> <http://nearbyhost.com:5984/jobhandlersims/>) failed:
> {worker_died,<0.9042.852>, {process_died,<0.19310.852>,
> {http_request_failed,"POST", "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs";,
> {error,{code,500}}}}}[error] 2019-04-21T12:39:27.995000Z
> couchdb@localhost <0.9322.852> -------- gen_server
> {couch_replicator_scheduler_job,{[54,99,99,99,99,98,48,50,97,55,52,56,
> 51,100,101,97,101,53,99,49,99,52,51,52,98,99,101,57,98,51,55,54],[43,9
> 9,111,110,116,105,110,117,111,117,115,43,99,114,101,97,116,101,95,116,
> 97,114,103,101,116]}}
> terminated with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}}
> last msg:
> {'EXIT',<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}}
> state:
> [{rep_id,{"6ccccb02a7483deae5c1c434bce9b376","+continuous+create_target"}},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{db_name,<<"shards/400000
> 00-5fffffff/_replicator.1526394302">>},{doc_id,<<"DE_to_US_for_jobhand
> lersims">>},{options,[{checkpoint_interval,30000},{connection_timeout,
> 30000},{continuous,true},{create_target,true},{http_connections,20},{r
> etries,5},{selector,{[{<<"$or">>,[{[{<<"type">>,{[{<<"$ne">>,<<"result
> s">>}]}}]},{[{<<"views">>,{[{<<"$gte">>,<<>>}]}}]},{[{<<"type">>,<<"re
> sults">>},{<<"requester_location">>,<<"US">>}]}]}]}},{socket_options,[
> {keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batc
> h_size,500},{worker_processes,4}]},{session_id,<<"436547a63366ea84a38d
> 8aeb36dd2d88">>},{start_seq,{0,<<"3194311-g1AAAAJ7eJzLYWBg4MhgTmEQTM4v
> Tc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAM
> kke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsZqDDTMwsUizMDIg1bAEkGH1cMNUDCGGmRqk
> mBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceA
> Bi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQD5VMZk">>}},{sourc
> e_seq,<<"3229426-g1AAAAFreJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJ
> TIkyf___z8riYHtnzQedUkKQDLJHqqUTQ6fUgeQ0nioUokz-JQmgJTWQ5WaLcSjNI8FSDI
> 0ACmg6vkg5R0PCSpfAFG-H6icNUKdoPIDEOX3QconXyCo_AFEOdjtdSVZAOiudpk">>},{
> committed_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzM
> nILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1L
> Mzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZG
> gAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1
> N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{current_through_seq
> ,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTI
> kyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB
> 5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMg
> HG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHE
> AOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{highest_seq_done,{8,<<"3204835-g
> 1AAAAI3eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge3
> T91ygGLu5qWFqimEyNj14TEpSAJJJ9iDDkhhYf17Cp9QBpDQebi-3MNheAxOLNAsjA1LtT
> QAZVg-xl01zKR6leSxAkqEBSAFVz4fY3egIttsy0Sgx2dKCRLshBi6AGLgfbCCrVxPYQKO
> 0REsLM2OyDDwAMfA-xMB2C0ismBslpyaakmXgA4iB0OAuSgcbaJKUkmxkmYhNaxYAmZez-
> A">>}}][error] 2019-04-21T12:39:27.995000Z couchdb@localhost
> <0.9322.852> -------- CRASH REPORT Process (<0.9322.852>) with 3
> neighbors exited with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}} at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors:
> [couch_replicator_scheduler_sup,couch_replicator_sup,...], messages:
> [{'EXIT',<0.12947.851>,{process_died,<0.9687.852>,...}},...], links:
> [<0.162.852>,<0.5422.852>,<0.10686.852>,<0.25534.850>,...], dictionary:
> [{pending_count_state,{{1555,850367,995000},9348}},{task_status_update
> ,...},...],
> trap_exit: true, status: running, heap_size: 10958, stack_size: 27,
> reductions: 63152[error] 2019-04-21T12:39:27.995000Z couchdb@localhost
> <0.10686.852> -------- gen_server <0.10686.852> terminated with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}}
> last msg:
> {'EXIT',<0.9322.852>,{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}}}
> state: [{main_pid,<0.9322.852>},{loop,<0.6805.852>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,1},{pending_
> fetch,nil},{batch_size,351185}][error]
> 2019-04-21T12:39:27.995000Z couchdb@localhost <0.10686.852> --------
> CRASH REPORT Process (<0.10686.852>) with 2 neighbors exited with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}} at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...]
> ,
> messages: [], links: [<0.6805.852>,<0.20253.851>], dictionary:
> [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status:
> running, heap_size: 2586, stack_size: 27, reductions: 496765[error]
> 2019-04-21T12:39:27.995000Z couchdb@localhost <0.385.0> --------
> Supervisor couch_replicator_scheduler_sup had child undefined started
> with {couch_replicator_scheduler_job,start_link,undefined} at
> <0.9322.852> exit with reason
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500
> }}}}} in context child_terminated[error] 2019-04-21T12:40:13.313000Z
> couchdb@localhost <0.23845.851> 7ec5e9ec3e rexi_server: from:
> couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3
> exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:40:13.423000Z couchdb@localhost
> <0.19545.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost
> <0.26362.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost
> <0.25940.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost
> <0.25878.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost
> <0.25863.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost
> <0.28372.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:40:13.438000Z couchdb@localhost
> <0.29759.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:43:23.323000Z couchdb@localhost
> <0.16895.850> ed368f9947
> rexi_server: from: couchdb@localhost(<0.29169.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{li
> ne,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}
> ]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{
> line,478}]}][error] 2019-04-21T12:43:23.339000Z couchdb@localhost
> <0.3976.852> ed368f9947
> rexi_server: from: couchdb@localhost(<0.29169.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},
> {couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{lin
> e,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]
> }][error] 2019-04-21T12:43:23.339000Z couchdb@localhost <0.3667.852>
> ed368f9947
> rexi_server: from: couchdb@localhost(<0.29169.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,chang
> es_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_en
> gine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},
> {couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848
> }]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,
> 775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},
> {couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{lin
> e,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]
> }]
>
