Hi Gustavo,
"no match of right hand value {error,eio}" looks like a failure to write to
a block device. Perhaps disks are full, or there is some kind of throttling
(over-quota) issue?
Replication will restart on failures. There is an exponential backoff on
repeated errors so if your replications crash too many times in a row
they'd be penalized and wait a bit. But I'd investigate why there are these
EIO errors happening first.
Cheers,
-Nick
On Mon, Apr 22, 2019 at 11:39 AM Delfino Gustavo <gustavo.delf...@zf.com>
wrote:
> I am doing multimaster replication where each server is configured to push
> its changes to the other one. The log I sent was from the remote server
> trying to send the changes to my local server. As an experiment I
> instructed the local server to pull from the remote and that is not working
> either. This is a segment of the log in the local server:
>
> [error] 2019-04-22T13:39:06.972000Z couchdb@localhost <0.20502.2490>
> -------- CRASH REPORT Process (<0.20502.2490>) with 1 neighbors exited
> with reason: no match of right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <= couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247)
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_stream,init,['Argument__1']}, ancestors: [<0.27672.2480>], messages:
> [], links: [<0.27672.2480>], dictionary:
> [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.15...">>}}],
> trap_exit: false, status: running, heap_size: 987, stack_size: 27,
> reductions: 596
> [error] 2019-04-22T13:39:07.081000Z couchdb@localhost <0.18062.2491>
> -------- Replicator, request PUT to "http://
> nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad07bc57-1:results?new_edits=false"
> failed due to error {code,500}
> [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.4458.2476>
> -------- Replicator, request PUT to "http://
> nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad0c5e1c-3:results?new_edits=false"
> failed due to error {code,500}
> [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490>
> -------- gen_server <0.22292.2490> terminated with reason: no match of
> right hand value {error,eio} at couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <= couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247)
> last msg: close
> state:
> {stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.74448900.31762>,[],[<<31,
> (...etc...),0>>],2186,4096,0,<<>>,<<>>,2186,#Fun<couch_stream.8.80129967>,#Fun<couch_stream.9.80129967>}
> [error] 2019-04-22T13:39:07.300000Z couchdb@localhost <0.22292.2490>
> -------- CRASH REPORT Process (<0.22292.2490>) with 1 neighbors exited
> with reason: no match of right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <= couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247)
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_stream,init,['Argument__1']}, ancestors: [<0.16555.2491>], messages:
> [], links: [<0.16555.2491>], dictionary:
> [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.15...">>}}],
> trap_exit: false, status: running, heap_size: 987, stack_size: 27,
> reductions: 676
> [error] 2019-04-22T13:39:07.627000Z couchdb@localhost <0.13178.2487>
> -------- Replicator, request PUT to "http://
> nearbyhost.com:5984/jobhandlersims/361c9af728e2775b4e611efcad043479-71:results?new_edits=false"
> failed due to error {code,500}
> [error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483>
> -------- gen_server <0.30019.2483> terminated with reason: no match of
> right hand value {error,eio} at couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <= couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247)
> last msg: close
> state:
> {stream,{couch_bt_engine_stream,{<0.10843.2485>,[]}},#Ref<0.0.88604680.223648>,[],[<<31,(...etc...),0>>],2039,4096,0,<<>>,<<>>,2039,#Fun<couch_stream.8.80129967>,#Fun<couch_stream.9.80129967>}
> [error] 2019-04-22T13:39:07.643000Z couchdb@localhost <0.30019.2483>
> -------- CRASH REPORT Process (<0.30019.2483>) with 1 neighbors exited
> with reason: no match of right hand value {error,eio} at
> couch_bt_engine_stream:write/2(line:60) <=
> couch_stream:do_write/2(line:302) <= couch_stream:handle_call/3(line:278)
> <= gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) <= proc_lib:init_p_do_apply/3(line:247)
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_stream,init,['Argument__1']}, ancestors: [<0.4711.2491>], messages:
> [], links: [<0.4711.2491>], dictionary:
> [{io_priority,{interactive,<<"shards/80000000-9fffffff/jobhandlersims.151...">>}}],
> trap_exit: false, status: running, heap_size: 987, stack_size: 27,
> reductions: 676
>
> I have now restarted the service and replication briefly resumed but it is
> now failing again. I'll continue to investigate.
>
> Regards,
>
> Gustavo Delfino
>
>
> -----Original Message-----
> From: Peter J Krawetzky <pkrawet...@gmail.com>
> Sent: Sunday, April 21, 2019 3:08 PM
> To: user@couchdb.apache.org
> Subject: Re: CouchDB replication crashing
>
> Are the errors below on the primary or the replication server? PeterSent
> from my Verizon, Samsung Galaxy smartphone
> -------- Original message --------From: Delfino Gustavo <
> gustavo.delf...@zf.com> Date: 4/21/19 13:25 (GMT-05:00) To:
> user@couchdb.apache.org Subject: CouchDB replication crashing Hello, I
> have had two CouchDB servers replicating a couple of DBs for several
> months. One server is local and the other is in remote location. Both are
> running CouchDB 2.2.0 under Windows 7. Replication is failing and I know I
> can reactivate it by restarting the remote server but I would like to
> understand if there is anything I can do to prevent this situation in the
> first place.In previous CouchDB versions, I used to have this problem where
> the replication seems to be disconnected from the network while the DB
> continues to be accessible. It had been working very well under 2.2.0. I
> wonder if moving up to 2.3 would help.A representative segment of the log
> file is found below.Regards,Gustavo Delfino[error]
> 2019-04-21T12:38:21.805000Z couchdb@localhost <0.28098.851> --------
> Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:38:24.020000Z
> couchdb@localhost <0.8055.852> -------- Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:38:28.497000Z
> couchdb@localhost <0.12289.852> -------- Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:38:51.819000Z
> couchdb@localhost <0.11004.852> -------- Replicator, request GET to "
> http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-211?revs=true&open_revs=%5B%221-4c79aa0880ec04131b40c06855b36ff4%22%5D&latest=true";
> failed due to error {error,req_timedout}[error] 2019-04-21T12:38:51.819000Z
> couchdb@localhost <0.9866.852> -------- Replicator, request GET to "
> http://farawayhost.com:5984/jobhandlersims/4bb0aa8269a879fdc95107973872dcfb-15?atts_since=%5B%222-597b62c56ad573a21a63e62c1117856e%22%5D&revs=true&open_revs=%5B%223-5f29e1678a58784ee2e8aee46c57b29d%22%5D&latest=true";
> failed due to error sel_conn_closed[error] 2019-04-21T12:38:52.240000Z
> couchdb@localhost <0.21284.852> -------- Replicator, request PUT to "
> http://nearbyhost.com:5984/jobhandlersims/f292cc7b8c282db3d41670a6e1411e31-120:results?new_edits=false";
> failed due to error {code,500}[error] 2019-04-21T12:39:01.678000Z
> couchdb@localhost <0.19310.852> -------- Replicator, request POST to "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to error
> {code,500}[error] 2019-04-21T12:39:01.678000Z couchdb@localhost
> <0.9042.852> -------- gen_server <0.9042.852> terminated with reason:
> {process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
> last msg: {'EXIT',<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
> state: [{main_pid,<0.9322.852>},{loop,<0.6467.852>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_fetch,nil},{batch_size,268141}][error]
> 2019-04-21T12:39:01.678000Z couchdb@localhost <0.9042.852> -------- CRASH
> REPORT Process (<0.9042.852>) with 1 neighbors exited with reason:
> {process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
> messages: [], links: [<0.9322.852>,<0.6467.852>], dictionary:
> [{last_stats_report,{1555,850299,901000}}], trap_exit: true, status:
> running, heap_size: 6772, stack_size: 27, reductions: 393291[error]
> 2019-04-21T12:39:18.916000Z couchdb@localhost <0.9687.852> --------
> Replicator, request POST to "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to error
> {code,500}[error] 2019-04-21T12:39:18.916000Z couchdb@localhost
> <0.12947.851> -------- gen_server <0.12947.851> terminated with reason:
> {process_died,<0.9687.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
> last msg: {'EXIT',<0.9687.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
> state: [{main_pid,<0.9322.852>},{loop,<0.15756.851>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,0},{pending_fetch,nil},{batch_size,383624}][error]
> 2019-04-21T12:39:18.916000Z couchdb@localhost <0.12947.851> --------
> CRASH REPORT Process (<0.12947.851>) with 1 neighbors exited with reason:
> {process_died,<0.9687.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
> messages: [], links: [<0.9322.852>,<0.15756.851>], dictionary:
> [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status:
> running, heap_size: 6772, stack_size: 27, reductions: 552772[error]
> 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> --------
> Replicator, request POST to "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs"; failed due to error
> {code,500}[error] 2019-04-21T12:39:27.964000Z couchdb@localhost
> <0.10607.852> -------- gen_server <0.10607.852> terminated with reason:
> {http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}
> at couch_replicator_httpc:report_error/4(line:363) <=
> couch_replicator_httpc:send_req/3(line:76) <=
> couch_replicator_worker:flush_docs/2(line:493) <=
> couch_replicator_worker:maybe_flush_docs/3(line:459) <=
> couch_replicator_worker:maybe_flush_docs/2(line:436) <=
> couch_replicator_worker:handle_call/3(line:124) <=
> gen_server:try_handle_call/4(line:615) <=
> gen_server:handle_msg/5(line:647) last msg:
> {batch_doc,{doc,<<"4bb0aa8269a879fdc95107973872dcfb-424">>,{3,[<<157,204,36,86,54,61,219,128,174,147,218,239,156,55,31,23>>,<<61,194,111,234,124,76,158,204,215,47,185,48,158,105,104,215>>,<<203,140,2,100,128,7,203,236,171,131,32,29,132,106,170,51>>]},
> (...erlang json removed...) ,[],false,[]}} state:
> [{main_pid,<0.9322.852>},{loop,<0.1165.852>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,20},{pending_fetch,{{<0.1165.852>,#Ref<0.0.23068677.99464>},{<<"4bb0aa8269a879fdc95107973872dcfb-174">>,[{3,<<139,102,22,196,97,193,111,166,119,37,58,206,57,73,182,130>>}],[{2,<<132,199,224,202,205,103,3,105,55,29,255,196,200,63,170,152>>}]}}},{batch_size,523788}][error]
> 2019-04-21T12:39:27.964000Z couchdb@localhost <0.10607.852> --------
> CRASH REPORT Process (<0.10607.852>) with 15 neighbors exited with reason:
> {http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
> messages:
> [{'EXIT',<0.24052.852>,normal},{'$gen_call',{<0.20083.852>,...},...},...],
> links: [<0.22451.852>,<0.24660.852>,<0.25927.852>,<0.27245.852>,...],
> dictionary:
> [{last_stats_report,{1555,850301,711000}},{ibrowse_stream_status,...}],
> trap_exit: true, status: running, heap_size: 28690, stack_size: 27,
> reductions: 744275[error] 2019-04-21T12:39:27.995000Z couchdb@localhost
> <0.9322.852> -------- Worker <0.9042.852> died with reason:
> {process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}[error]
> 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> --------
> Replication `6ccccb02a7483deae5c1c434bce9b376+continuous+create_target` (`
> http://farawayhost.com:5984/jobhandlersims/`
> <http://farawayhost.com:5984/jobhandlersims/> -> `
> http://nearbyhost.com:5984/jobhandlersims/`
> <http://nearbyhost.com:5984/jobhandlersims/>) failed:
> {worker_died,<0.9042.852>, {process_died,<0.19310.852>,
> {http_request_failed,"POST", "
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs";,
> {error,{code,500}}}}}[error] 2019-04-21T12:39:27.995000Z couchdb@localhost
> <0.9322.852> -------- gen_server
> {couch_replicator_scheduler_job,{[54,99,99,99,99,98,48,50,97,55,52,56,51,100,101,97,101,53,99,49,99,52,51,52,98,99,101,57,98,51,55,54],[43,99,111,110,116,105,110,117,111,117,115,43,99,114,101,97,116,101,95,116,97,114,103,101,116]}}
> terminated with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
> last msg:
> {'EXIT',<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
> state:
> [{rep_id,{"6ccccb02a7483deae5c1c434bce9b376","+continuous+create_target"}},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{db_name,<<"shards/40000000-5fffffff/_replicator.1526394302">>},{doc_id,<<"DE_to_US_for_jobhandlersims">>},{options,[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{create_target,true},{http_connections,20},{retries,5},{selector,{[{<<"$or">>,[{[{<<"type">>,{[{<<"$ne">>,<<"results">>}]}}]},{[{<<"views">>,{[{<<"$gte">>,<<>>}]}}]},{[{<<"type">>,<<"results">>},{<<"requester_location">>,<<"US">>}]}]}]}},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}]},{session_id,<<"436547a63366ea84a38d8aeb36dd2d88">>},{start_seq,{0,<<"3194311-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsZqDDTMwsUizMDIg1bAEkGH1cMNUDCGGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQD5VMZk">>}},{source_seq,<<"3229426-g1AAAAFreJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8riYHtnzQedUkKQDLJHqqUTQ6fUgeQ0nioUokz-JQmgJTWQ5WaLcSjNI8FSDI0ACmg6vkg5R0PCSpfAFG-H6icNUKdoPIDEOX3QconXyCo_AFEOdjtdSVZAOiudpk">>},{committed_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{current_through_seq,{2,<<"3194325-g1AAAAJ7eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge2dWC5QjN3c1DA1xTAZmx48JiUpAMkke5hhrJ8_gA1LMzczs7C0JNUwB5Bh8XCXsbqADTMwsUizMDIg1bAEkGH1cMNUjCCGmRqkmBkak2hYHguQZGgAUkDz5kMMbMgHG2iZaJSYbGlBloELIAbuh4SdSxDYQKO0REsLM_JceABi4H2IgY1MkJg1N0pOTTQly8AHEAOhYZgzDWygSVJKspFlIjatWQAQ4MZy">>}},{highest_seq_done,{8,<<"3204835-g1AAAAI3eJzLYWBg4MhgTmEQTM4vTc5ISXLIyU9OzMnILy7JAUoxJTIkyf___z8rgzmJge3T91ygGLu5qWFqimEyNj14TEpSAJJJ9iDDkhhYf17Cp9QBpDQebi-3MNheAxOLNAsjA1LtTQAZVg-xl01zKR6leSxAkqEBSAFVz4fY3egIttsy0Sgx2dKCRLshBi6AGLgfbCCrVxPYQKO0REsLM2OyDDwAMfA-xMB2C0ismBslpyaakmXgA4iB0OAuSgcbaJKUkmxkmYhNaxYAmZez-A">>}}][error]
> 2019-04-21T12:39:27.995000Z couchdb@localhost <0.9322.852> -------- CRASH
> REPORT Process (<0.9322.852>) with 3 neighbors exited with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_scheduler_job,init,['Argument__1']}, ancestors:
> [couch_replicator_scheduler_sup,couch_replicator_sup,...], messages:
> [{'EXIT',<0.12947.851>,{process_died,<0.9687.852>,...}},...], links:
> [<0.162.852>,<0.5422.852>,<0.10686.852>,<0.25534.850>,...], dictionary:
> [{pending_count_state,{{1555,850367,995000},9348}},{task_status_update,...},...],
> trap_exit: true, status: running, heap_size: 10958, stack_size: 27,
> reductions: 63152[error] 2019-04-21T12:39:27.995000Z couchdb@localhost
> <0.10686.852> -------- gen_server <0.10686.852> terminated with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
> last msg:
> {'EXIT',<0.9322.852>,{worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}}
> state: [{main_pid,<0.9322.852>},{loop,<0.6805.852>},{source,"
> http://farawayhost.com:5984/jobhandlersims/"},{target,";
> http://nearbyhost.com:5984/jobhandlersims/"},{num_readers,1},{pending_fetch,nil},{batch_size,351185}][error]
> 2019-04-21T12:39:27.995000Z couchdb@localhost <0.10686.852> --------
> CRASH REPORT Process (<0.10686.852>) with 2 neighbors exited with reason:
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
> at gen_server:terminate/7(line:812) <=
> proc_lib:init_p_do_apply/3(line:247); initial_call:
> {couch_replicator_worker,init,['Argument__1']}, ancestors:
> [<0.9322.852>,couch_replicator_scheduler_sup,couch_replicator_sup,...],
> messages: [], links: [<0.6805.852>,<0.20253.851>], dictionary:
> [{last_stats_report,{1555,850299,839000}}], trap_exit: true, status:
> running, heap_size: 2586, stack_size: 27, reductions: 496765[error]
> 2019-04-21T12:39:27.995000Z couchdb@localhost <0.385.0> --------
> Supervisor couch_replicator_scheduler_sup had child undefined started with
> {couch_replicator_scheduler_job,start_link,undefined} at <0.9322.852> exit
> with reason
> {worker_died,<0.9042.852>,{process_died,<0.19310.852>,{http_request_failed,"POST","
> http://nearbyhost.com:5984/jobhandlersims/_bulk_docs",{error,{code,500}}}}}
> in context child_terminated[error] 2019-04-21T12:40:13.313000Z
> couchdb@localhost <0.23845.851> 7ec5e9ec3e rexi_server: from:
> couchdb@localhost(<0.28700.851>) mfa: fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:40:13.423000Z couchdb@localhost <0.19545.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:40:13.438000Z couchdb@localhost <0.26362.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:40:13.438000Z couchdb@localhost <0.25940.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:40:13.438000Z couchdb@localhost <0.25878.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:40:13.438000Z couchdb@localhost <0.25863.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:40:13.438000Z couchdb@localhost <0.28372.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:40:13.438000Z couchdb@localhost <0.29759.851> 7ec5e9ec3e
> rexi_server: from: couchdb@localhost(<0.28700.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:43:23.323000Z couchdb@localhost <0.16895.850> ed368f9947
> rexi_server: from: couchdb@localhost(<0.29169.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,stream_kp_node,8,[{file,"src/couch_btree.erl"},{line,819}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]}][error]
> 2019-04-21T12:43:23.339000Z couchdb@localhost <0.3976.852> ed368f9947
> rexi_server: from: couchdb@localhost(<0.29169.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]}][error]
> 2019-04-21T12:43:23.339000Z couchdb@localhost <0.3667.852> ed368f9947
> rexi_server: from: couchdb@localhost(<0.29169.851>) mfa:
> fabric_rpc:changes/3 exit:timeout
> [{rexi,stream2,3,[{file,"src/rexi.erl"},{line,213}]},{fabric_rpc,changes_enumerator,2,[{file,"src/fabric_rpc.erl"},{line,380}]},{couch_bt_engine,drop_reductions,4,[{file,"src/couch_bt_engine.erl"},{line,903}]},{couch_btree,stream_kv_node2,8,[{file,"src/couch_btree.erl"},{line,848}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,222}]},{couch_bt_engine,fold_changes,5,[{file,"src/couch_bt_engine.erl"},{line,478}]},{fabric_rpc,changes,4,[{file,"src/fabric_rpc.erl"},{line,85}]}]
>
