Hi all!! I've been observing logs for a while and trying to understand why a few errors happen while (pull) replication is happening and also trying to understand the consequences.
** connection_closed* Log: ``` [error] <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2210613426-g1AAAAOCeJyd0r1OwzAQB3CrRYIHYC8FCSZS2_kwEUM7svURiu1zVKqQSlUZYKFP0FeAV2DsBE_BCm-Suj4UKqRYIctZsv66n053OSHkaNoFcgpKzxdmBIrRQEt1lz1SygKdzx9AFsugMMvcZjuSqF5ZlrNpV5Hj8-t7-3eYZJyDzoD0qyZhTY8_lPBR6sRWNay0_pvTmJA8ScS-xhppPPJqo502qbSbwGlUUwNXrMVsiVe73WnPlXbx5DSQXBihW8xWF3NacWArWdnHgi8oXn7ifGHEeST_L_rPBMVXFN9RHORO5FowIeKm4m_KeywIfiD4heD6zIFpbCAL4xYrrIvtid8o_qxxvHGiiVKTQqM1zrad2hfx%22&timeout=10000"; failed due to error connection_closed ``` Cause: This one happens from time to time. It looks like the source is closing connection, but why? Consequences: None. I think replicator just retries the task. * *changes_reader_died* Log: ``` [error] <0.15968.1946> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000"; failed due to error connection_closed [error] <0.13952.1946> -------- ChangesReader process died with reason: {http_request_failed,"GET"," https://admin:*****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed }} [error] <0.13952.1946> -------- Replication `363ed31e35361d024d22e0a990ca1ae5+continuous` (`https://admin:*****@source.ip:443/my_db/` -> `http://admin:*****@127.0.0.1:5984/my_db/`) failed: changes_reader_died [error] <0.30404.6252> -------- Replicator, request GET to "https://admin:*****@source.ip:443/my_db/b666ee7ec8ed5b82ae035fe5fa6a4a2c?revs=true&open_revs=%5B%221-5a7ba8a8658bd0ab7b3656aa089aa183%22%5D&latest=true"; failed due to error {error,closing_on_request} [error] <0.352.0> -------- Error in replication `363ed31e35361d024d22e0a990ca1ae5+continuous` (triggered by document `28d973af08931f79fa423b54a8004b31`): changes_reader_died Restarting replication in 5 seconds. [error] <0.13952.1946> -------- gen_server <0.13952.1946> terminated with reason: changes_reader_died last msg: {'EXIT',<0.15968.1946>,{http_request_failed,"GET","https://admin: *****@source.ip:443/my_db/_changes?feed=continuous&style=all_docs&since=%2235690238-g1...&timeout=10000",{error,connection_closed}}} state: [{data,[{"State",{rep_state,{rep,{"363ed31e35361d024d22e0a990ca1ae5","+continuous"},{httpdb,"https://admin: *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,nil,20},{httpdb,"http://admin: *****@127.0.0.1:5984/my_db/ ",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,nil,20},[{checkpoint_interval,30000},{connection_timeout,30000},{continuous,true},{http_connections,20},{retries,10},{socket_options,[{keepalive,true},{nodelay,false}]},{use_checkpoints,true},{worker_batch_size,500},{worker_processes,4}],{user_ctx,null,[],undefined},db,nil,<<"28d973af08931f79fa423b54a8004b31">>,<<"shards/c0000000-dfffffff/_replicator.1501167373">>},"https://admin: *****@source.ip:443/my_db/","http://admin:*****@127.0.0.1:5984/my_db/ ",{httpdb,"https://admin: *****@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin: *****@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[],{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"source_last_seq">>,<<"36567936-g1...">>},{<<"replication_id_version">>,3},{<<"history">>,[{[{<<"session_id">>,<<"6f7aa01e879f5e1841cdb7100f40806b">>},{<<"start_time">>,<<"Fri, 28 Jul 2017 09:46:48 GMT">>},{<<"end_time">>,<<"Sun, 30 Jul 2017 02:28:59 GMT">>},{<<"start_last_seq">>,0},{<<"end_last_seq">>,<<"36567936-g1...">>},{<<"recorded_seq">>,<<"36567936-g1...">>},{<<"missing_checked">>,22991404},{<<"missing_found">>,22991404},{<<"docs_read">>,22991404},{<<"docs_written">>,22991404},{<<"doc_write_failures">>,0}]}]}]},{0,0},{46381,<<"36567936-g1...">>},{46388,<<"36574076-g1...">>},[{46389,<<"36575175-g1...">>},{46390,<<"36575938-g1...">>},{46391,<<"36576695-...">>},{46392,<<"36577391-g1...">>}],{46388,<<"36574076-g1...">>},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},{doc,<<"_local/363ed31e35361d024d22e0a990ca1ae5">>,{0,[<<"4647">>]},{[]},[],false,[]},"Fri, 28 Jul 2017 09:46:48 GMT",<<"0">>,<<"0">>,nil,<0.16590.1946>,<0.16953.1946>,<0.15968.1946>,[<0.18469.1946>,<0.10009.1946>,<0.16617.1946>,<0.16155.1946>],[{doc_write_failures,0},{docs_read,22994904},{docs_written,22994904},{missing_checked,22994904},{missing_found,22994904}],<<"6f7aa01e879f5e1841cdb7100f40806b">>,nil,nil,nil,nil,<<"255587270-g1...">>,true,30000,db,nil}}]}] [error] <0.13952.1946> -------- CRASH REPORT Process (<0.13952.1946>) with 2 neighbors exited with reason: changes_reader_died at gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_replicator,init,['Argument__1']}, ancestors: [couch_replicator_job_sup,couch_replicator_sup,<0.349.0>], messages: [], links: [<0.16590.1946>,<0.16953.1946>,<0.18469.1946>,<0.16617.1946>,...], dictionary: [{task_status_props,[{changes_pending,127404519},{checkpoint_interval,...},...]},...], trap_exit: true, status: running, heap_size: 4185, stack_size: 27, reductions: 26502159 [error] <0.364.0> -------- Supervisor couch_replicator_job_sup had child "363ed31e35361d024d22e0a990ca1ae5+continuous" started with {gen_server,start_link,undefined} at <0.13952.1946> exit with reason changes_reader_died in context child_terminated [error] <0.16617.1946> -------- gen_server <0.16617.1946> terminated with reason: changes_reader_died last msg: {'EXIT',<0.13952.1946>,changes_reader_died} state: {state,<0.13952.1946>,<0.15004.1946>,20,{httpdb," https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.2293.6254>,<0.19041.6253>,<0.19785.6253>,<0.7470.6253>,<0.23117.6253>,<0.31026.6250>,<0.11251.6253>,<0.27821.6253>,<0.8477.6253>,<0.28980.6253>,<0.22848.6253>,<0.16549.6251>,<0.26208.6253>,<0.27365.6253>,<0.15830.6253>,<0.24330.6253>,<0.20562.6253>,<0.20869.6253>,<0.21799.6253>,<0.16311.6253>],nil,{{<0.15004.1946>,#Ref<0.0.6636.248998>},{<<"07597cb4d450c2b6ff674b675b9d5168">>,[{1,<<93,162,57,7,69,142,28,226,77,41,199,156,216,131,208,152>>}],[]}},nil,[{docs_read,199},{docs_written,25},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"b65rnw8pcm0jbt7jdpxvgmo1r\",\"_rev\":\"2-4d5283c4d125a0da73b95cc80b92abd6\ ",...}} [error] <0.16155.1946> -------- gen_server <0.16155.1946> terminated with reason: changes_reader_died last msg: {'EXIT',<0.13952.1946>,changes_reader_died} state: {state,<0.13952.1946>,<0.21653.1946>,20,{httpdb," https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.27802.6253>,<0.27692.6253>,<0.10519.6253>,<0.5584.6253>,<0.9485.6253>,<0.2597.6253>,<0.12901.6253>,<0.21982.6253>,<0.20758.6253>,<0.12367.6253>,<0.30450.6253>,<0.15900.6253>,<0.17133.6253>,<0.22945.6253>,<0.8889.6253>,<0.20333.6253>,<0.12924.6253>,<0.32204.6252>,<0.32140.6253>,<0.5556.6253>],nil,{{<0.21653.1946>,#Ref<0.0.6636.251466>},{<<"9lcen85eu5ztrg8cs287sacoy">>,[{2,<<58,122,97,121,57,131,34,12,88,18,7,165,20,247,0,70>>}],[]}},nil,[{docs_read,436},{docs_written,57},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"1802v33hahs2ql9951vitjbih\",\"_rev\":\"5-138167bbafe87da23205b457c3918bd4\ ",...],...}} [error] <0.16617.1946> -------- CRASH REPORT Process (<0.16617.1946>) with 15 neighbors exited with reason: changes_reader_died at gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], messages: [], links: [<0.21799.6253>,<0.24330.6253>,<0.27365.6253>,<0.28980.6253>,...], dictionary: [{ibrowse_stream_status,init},{last_stats_report,{1501,381757,...}}], trap_exit: true, status: running, heap_size: 4185, stack_size: 27, reductions: 6921530028 [error] <0.16155.1946> -------- CRASH REPORT Process (<0.16155.1946>) with 15 neighbors exited with reason: changes_reader_died at gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], messages: [], links: [<0.12924.6253>,<0.21982.6253>,<0.27692.6253>,<0.30450.6253>,...], dictionary: [{ibrowse_stream_status,init},{last_stats_report,{1501,381751,...}}], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 6914251593 [error] <0.18469.1946> -------- gen_server <0.18469.1946> terminated with reason: changes_reader_died last msg: {'EXIT',<0.13952.1946>,changes_reader_died} state: {state,<0.13952.1946>,<0.21935.1946>,20,{httpdb," https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@127.0.0.1:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.12414.6254>,<0.24987.6253>,<0.24685.6253>,<0.30475.6253>,<0.25412.6253>,<0.18441.6253>,<0.23627.6253>,<0.10529.6252>,<0.18425.6253>,<0.18994.6250>,<0.2214.6253>,<0.9544.6253>,<0.13738.6253>,<0.32462.6253>,<0.15946.6253>,<0.25304.6253>,<0.16690.6253>,<0.3468.6253>,<0.17659.6253>,<0.13537.6253>],nil,{{<0.21935.1946>,#Ref<0.0.6636.248486>},{<<"935cd70fbe4e02cb374180daa08a9fff">>,[{2,<<194,192,159,166,5,0,237,72,59,193,108,4,228,198,91,95>>}],[]}},nil,[{missing_checked,500},{missing_found,500}],nil,nil,{batch,[],0 }} [error] <0.10009.1946> -------- gen_server <0.10009.1946> terminated with reason: changes_reader_died last msg: {'EXIT',<0.13952.1946>,changes_reader_died} state: {state,<0.13952.1946>,<0.22823.1946>,20,{httpdb," https://admin:******@source.ip:443/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\ <https://admin:******@source.ip:443/maxi_billing/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{is_ssl,true},{socket_options,[{keepalive,true},{nodelay,false}]},{ssl_options,[{depth,3},{verify,0}]}],10,250,<0.1722.1946>,20},{httpdb,"http://admin:******@source.ip:5984/my_db/",nil,[{"Accept","application/json"},{"User-Agent","CouchDB-Replicator/2.0.0"}],30000,[{socket_options,[{keepalive,true},{nodelay,false}]}],10,250,<0.21751.1946>,20},[<0.19696.6253>,<0.2504.6253>,<0.13314.6253>,<0.31996.6253>,<0.18686.6253>,<0.20563.6253>,<0.23938.6253>,<0.8048.6253>,<0.18021.6253>,<0.31079.6253>,<0.19418.6253>,<0.18053.6253>,<0.22230.6253>,<0.15505.6253>,<0.14240.6253>,<0.10981.6253>,<0.29957.6253>,<0.13571.6253>,<0.17024.6253>,<0.29113.6253>],nil,{{<0.22823.1946>,#Ref<0.0.6636.249314>},{<<"31wv3qb6ipiql29u3t1nbmh4p">>,[{2,<<129,194,93,245,75,131,59,18,8,235,62,169,28,55,201,79>>}],[]}},nil,[{docs_read,140},{docs_written,18},{missing_checked,500},{missing_found,500}],nil,nil,{batch,[<<"{\"_id\":\"20ab5179602db3c31abb60507b40ebe9\",\"_rev\":\"2-d774bbcec2927a9c2de7f24a21486765\> ",,...],...}} [error] <0.18469.1946> -------- CRASH REPORT Process (<0.18469.1946>) with 15 neighbors exited with reason: changes_reader_died at gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], messages: [], links: [<0.18441.6253>,<0.25304.6253>,<0.30475.6253>,<0.32462.6253>,...], dictionary: [{ibrowse_stream_status,init},{last_stats_report,{1501,381762,...}}], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 6918077761 [error] <0.10009.1946> -------- CRASH REPORT Process (<0.10009.1946>) with 15 neighbors exited with reason: changes_reader_died at gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_replicator_worker,init,['Argument__1']}, ancestors: [<0.13952.1946>,couch_replicator_job_sup,couch_replicator_sup,...], messages: [], links: [<0.18053.6253>,<0.23938.6253>,<0.29957.6253>,<0.31079.6253>,...], dictionary: [{ibrowse_stream_status,init},{last_stats_report,{1501,381758,...}}], trap_exit: true, status: running, heap_size: 6772, stack_size: 27, reductions: 6919159795 ``` Cause: This one looks similar to the one before. The trigger seems to be that the source closes the connection for some reason, but the logs are much more verbose this time. Consequences: None. I think replicator just retries the task. Furthermore, this last logs show the username and passwords plain text on the state key. Sorry for the wall of text. Regards -- [image: Cabify - Your private Driver] <http://www.cabify.com/> *Carlos Alonso* Data Engineer Madrid, Spain [email protected] Prueba gratis con este código #CARLOSA6319 <https://cabify.com/i/carlosa6319> [image: Facebook] <http://cbify.com/fb_ES>[image: Twitter] <http://cbify.com/tw_ES>[image: Instagram] <http://cbify.com/in_ES>[image: Linkedin] <https://www.linkedin.com/in/mrcalonso> -- Este mensaje y cualquier archivo adjunto va dirigido exclusivamente a su destinatario, pudiendo contener información confidencial sometida a secreto profesional. No está permitida su reproducción o distribución sin la autorización expresa de Cabify. Si usted no es el destinatario final por favor elimínelo e infórmenos por esta vía. This message and any attached file are intended exclusively for the addressee, and it may be confidential. You are not allowed to copy or disclose it without Cabify's prior written authorization. If you are not the intended recipient please delete it from your system and notify us by e-mail.
