bucardo went down last night. On startup I got the following in the
bucardo log:
tail -f /var/log/bucardo/log.bucardo
(17758) [Tue Mar 31 09:24:28 2020] MCP Found stopfile
"/var/run/bucardo/fullstopbucardo": exiting
(18055) [Tue Mar 31 09:24:28 2020] CTL Found stopfile
"/var/run/bucardo/fullstopbucardo": exiting
(18055) [Tue Mar 31 09:24:28 2020] CTL Warning! Controller for
"cargotel_common_sync" was killed at line 2192: Found stopfile: stop |
(18056) [Tue Mar 31 09:24:28 2020] CTL Found stopfile
"/var/run/bucardo/fullstopbucardo": exiting
(18056) [Tue Mar 31 09:24:28 2020] CTL Warning! Controller for
"load_sync" was killed at line 2192: Found stopfile: stop |
(18079) [Tue Mar 31 09:24:28 2020] KID (cargotel_common_sync) Found
stopfile "/var/run/bucardo/fullstopbucardo": exiting
(18079) [Tue Mar 31 09:24:28 2020] KID (cargotel_common_sync) Kid 18079
exiting at cleanup_kid. Reason: Normal exit
(18055) [Tue Mar 31 09:24:29 2020] CTL Controller 18055 exiting at
cleanup_controller. Reason: Found stopfile: stop |
(18056) [Tue Mar 31 09:24:29 2020] CTL Controller 18056 exiting at
cleanup_controller. Reason: Found stopfile: stop |
(17758) [Tue Mar 31 09:24:30 2020] MCP End of cleanup_mcp. Sys time: Tue
Mar 31 09:24:30 2020. Database time: 2020-03-31 09:24:30.263517-04
(17758) [Tue Mar 31 09:24:30 2020] MCP Exiting
*** Error in `Bucardo VAC.': double free or corruption (!prev):
0x00000000215ec990 ***
======= Backtrace: =========
/usr/lib64/libc.so.6(+0x7c503)[0x7fb60ba9e503]
/usr/lib64/perl5/vendor_perl/auto/DBD/Pg/Pg.so(pg_st_destroy+0x16f)[0x7fb603a67b3f]
/usr/lib64/perl5/vendor_perl/auto/DBD/Pg/Pg.so(+0xc2aa)[0x7fb603a532aa]
/usr/lib64/perl5/vendor_perl/auto/DBI/DBI.so(XS_DBI_dispatch+0xc47)[0x7fb6050e9707]
/usr/lib64/perl5/CORE/libperl.so(Perl_pp_entersub+0x58f)[0x7fb60ce2e42f]
/usr/lib64/perl5/CORE/libperl.so(Perl_call_sv+0x69d)[0x7fb60cdbe41d]
/usr/lib64/perl5/CORE/libperl.so(+0xc5185)[0x7fb60ce37185]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x4b0)[0x7fb60ce37aa0]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x4a)[0x7fb60ce3810a]
/usr/lib64/perl5/CORE/libperl.so(+0xbd887)[0x7fb60ce2f887]
/usr/lib64/perl5/CORE/libperl.so(Perl_sv_clean_objs+0x26)[0x7fb60ce38586]
/usr/lib64/perl5/CORE/libperl.so(perl_destruct+0x14be)[0x7fb60cdc08de]
Bucardo VAC.[0x400d61]
/usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fb60ba43b35]
Bucardo VAC.[0x400dd1]
======= Memory map: ========
00400000-00402000 r-xp 00000000 fd:03 105121721
/usr/bin/perl
00601000-00602000 r--p 00001000 fd:03 105121721
/usr/bin/perl
00602000-00603000 rw-p 00002000 fd:03 105121721
/usr/bin/perl
01861000-0c9d1000 rw-p 00000000 00:00 0
[heap]
0c9d1000-2407b000 rw-p 00000000 00:00 0
[heap]
7fb5f8000000-7fb5f8021000 rw-p 00000000 00:00 0
7fb5f8021000-7fb5fc000000 ---p 00000000 00:00 0
7fb5feec2000-7fb5ff4c4000 rw-p 00000000 00:00 0
7fb5ff6c5000-7fb5ff6ca000 r-xp 00000000 fd:03 105387569
/usr/lib64/libnss_dns-2.17.so
7fb5ff6ca000-7fb5ff8c9000 ---p 00005000 fd:03 105387569
/usr/lib64/libnss_dns-2.17.so
7fb5ff8c9000-7fb5ff8ca000 r--p 00004000 fd:03 105387569
/usr/lib64/libnss_dns-2.17.so
7fb5ff8ca000-7fb5ff8cb000 rw-p 00005000 fd:03 105387569
/usr/lib64/libnss_dns-2.17.so
7fb5ff8cb000-7fb5ff8ce000 r-xp 00000000 fd:03 105385581
/usr/lib64/perl5/auto/MIME/Base64/Base64.so
7fb5ff8ce000-7fb5ffacd000 ---p 00003000 fd:03 105385581
/usr/lib64/perl5/auto/MIME/Base64/Base64.so
7fb5ffacd000-7fb5fface000 r--p 00002000 fd:03 105385581
/usr/lib64/perl5/auto/MIME/Base64/Base64.so
7fb5fface000-7fb5ffacf000 rw-p 00003000 fd:03 105385581
/usr/lib64/perl5/auto/MIME/Base64/Base64.so
7fb5ffacf000-7fb5ffad2000 r-xp 00000000 fd:03 105390713
/usr/lib64/perl5/vendor_perl/auto/Sys/Syslog/Syslog.so
7fb5ffad2000-7fb5ffcd2000 ---p 00003000 fd:03 105390713
/usr/lib64/perl5/vendor_perl/auto/Sys/Syslog/Syslog.so
7fb5ffcd2000-7fb5ffcd3000 r--p 00003000 fd:03 105390713
/usr/lib64/perl5/vendor_perl/auto/Sys/Syslog/Syslog.so
7fb5ffcd3000-7fb5ffcd4000 rw-p 00004000 fd:03 105390713
/usr/lib64/perl5/vendor_perl/auto/Sys/Syslog/Syslog.so
7fb5ffcd4000-7fb5ffcdc000 r-xp 00000000 fd:03 105385282
/usr/lib64/perl5/vendor_perl/auto/Socket/Socket.so
7fb5ffcdc000-7fb5ffedc000 ---p 00008000 fd:03 105385282
/usr/lib64/perl5/vendor_perl/auto/Socket/Socket.so
7fb5ffedc000-7fb5ffede000 r--p 00008000 fd:03 105385282
/usr/lib64/perl5/vendor_perl/auto/Socket/Socket.so
7fb5ffede000-7fb5ffedf000 rw-p 0000a000 fd:03 105385282
/usr/lib64/perl5/vendor_perl/auto/Socket/Socket.so
7fb5ffedf000-7fb5ffee0000 r-xp 00000000 fd:03 105385624
/usr/lib64/perl5/auto/Sys/Hostname/Hostname.so
7fb5ffee0000-7fb6000df000 ---p 00001000 fd:03 105385624
/usr/lib64/perl5/auto/Sys/Hostname/Hostname.so
7fb6000df000-7fb6000e0000 r--p 00000000 fd:03 105385624
/usr/lib64/perl5/auto/Sys/Hostname/Hostname.so
7fb6000e0000-7fb6000e1000 rw-p 00001000 fd:03 105385624
/usr/lib64/perl5/auto/Sys/Hostname/Hostname.so
7fb6001cd000-7fb6001e2000 r-xp 00000000 fd:03 105389922
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fb6001e2000-7fb6003e1000 ---p 00015000 fd:03 105389922
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fb6003e1000-7fb6003e2000 r--p 00014000 fd:03 105389922
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fb6003e2000-7fb6003e3000 rw-p 00015000 fd:03 105389922
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7fb6003e3000-7fb600464000 rw-p 00000000 00:00 0
7fb600464000-7fb600470000 r-xp 00000000 fd:03 105384954
/usr/lib64/libnss_files-2.17.so
7fb600470000-7fb60066f000 ---p 0000c000 fd:03 105384954
/usr/lib64/libnss_files-2.17.so
7fb60066f000-7fb600670000 r--p 0000b000 fd:03 105384954
/usr/lib64/libnss_files-2.17.so
7fb600670000-7fb600671000 rw-p 0000c000 fd:03 105384954
/usr/lib64/libnss_files-2.17.so
7fb600671000-7fb600677000 rw-p 00000000 00:00 0
7fb600677000-7fb6006d7000 r-xp 00000000 fd:03 105387562
/usr/lib64/libpcre.so.1.2.0
7fb6006d7000-7fb6008d6000 ---p 00060000 fd:03 105387562
/usr/lib64/libpcre.so.1.2.0
7fb6008d6000-7fb6008d7000 r--p 0005f000 fd:03 105387562
/usr/lib64/libpcre.so.1.2.0
7fb6008d7000-7fb6008d8000 rw-p 00060000 fd:03 105387562
/usr/lib64/libpcre.so.1.2.0
7fb6008d8000-7fb6008fc000 r-xp 00000000 fd:03 105384667
/usr/lib64/libselinux.so.1
7fb6008fc000-7fb600afb000 ---p 00024000 fd:03 105384667
/usr/lib64/libselinux.so.1
7fb600afb000-7fb600afc000 r--p 00023000 fd:03 105384667
/usr/lib64/libselinux.so.1
7fb600afc000-7fb600afd000 rw-p 00024000 fd:03 105384667
/usr/lib64/libselinux.so.1
7fb600afd000-7fb600aff000 rw-p 00000000 00:00 0
7fb600aff000-7fb600b39000 r-xp 00000000 fd:03 105390252
/usr/lib64/libnspr4.so
7fb600b39000-7fb600d38000 ---p 0003a000 fd:03 105390252
/usr/lib64/libnspr4.so
7fb600d38000-7fb600d39000 r--p 00039000 fd:03 105390252
/usr/lib64/libnspr4.so
7fb600d39000-7fb600d3b000 rw-p 0003a000 fd:03 105390252
/usr/lib64/libnspr4.so
7fb600d3b000-7fb600d3d000 rw-p 00000000 00:00 0
7fb600d3d000-7fb600d41000 r-xp 00000000 fd:03 105390253
/usr/lib64/libplc4.so
7fb600d41000-7fb600f40000 ---p 00004000 fd:03 105390253
/usr/lib64/libplc4.so
7fb600f40000-7fb600f41000 r--p 00003000 fd:03 105390253
/usr/lib64/libplc4.so
7fb600f41000-7fb600f42000 rw-p 00004000 fd:03 105390253
/usr/lib64/libplc4.so
7fb600f42000-7fb600f45000 r-xp 00000000 fd:03 105390254
/usr/lib64/libplds4.so
7fb600f45000-7fb601144000 ---p 00003000 fd:03 105390254
/usr/lib64/libplds4.so
7fb601144000-7fb601145000 r--p 00002000 fd:03 105390254
/usr/lib64/libplds4.so
7fb601145000-7fb601146000 rw-p 00003000 fd:03 105390254
/usr/lib64/libplds4.so
7fb601146000-7fb60116c000 r-xp 00000000 fd:03 105390255
/usr/lib64/libnssutil3.so
7fb60116c000-7fb60136b000 ---p 00026000 fd:03 105390255
/usr/lib64/libnssutil3.so
7fb60136b000-7fb601372000 r--p 00025000 fd:03 105390255
/usr/lib64/libnssutil3.so
7fb601372000-7fb601373000 rw-p 0002c000 fd:03 105390255
/usr/lib64/libnssutil3.so
7fb601373000-7fb601494000 r-xp 00000000 fd:03 105390262
/usr/lib64/libnss3.so
7fb601494000-7fb601694000 ---p 00121000 fd:03 105390262
/usr/lib64/libnss3.so
7fb601694000-7fb601699000 r--p 00121000 fd:03 105390262
/usr/lib64/libnss3.so
7fb601699000-7fb60169b000 rw-p 00126000 fd:03 105390262
/usr/lib64/libnss3.so
7fb60169b000-7fb60169d000 rw-p 00000000 00:00 0
7fb60169d000-7fb6016c1000 r-xp 00000000 fd:03 105390263
/usr/lib64/libsmime3.so
7fb6016c1000-7fb6018c0000 ---p 00024000 fd:03 105390263
/usr/lib64/libsmime3.so
7fb6018c0000-7fb6018c3000 r--p 00023000 fd:03 105390263
/usr/lib64/libsmime3.so
7fb6018c3000-7fb6018c4000 rw-p 00026000 fd:03 105390263
/usr/lib64/libsmime3.so
7fb6018c4000-7fb60190b000 r-xp 00000000 fd:03 105390264
/usr/lib64/libssl3.so
7fb60190b000-7fb601b0a000 ---p 00047000 fd:03 105390264
/usr/lib64/libssl3.so
7fb601b0a000-7fb601b0e000 r--p 00046000 fd:03 105390264
/usr/lib64/libssl3.so
7fb601b0e000-7fb601b0f000 rw-p 0004a000 fd:03 105390264
/usr/lib64/libssl3.so
7fb601b0f000-7fb601b10000 rw-p 00000000 00:00 0
7fb601b10000-7fb601b2c000 r-xp 00000000 fd:03 105384823
/usr/lib64/libsasl2.so.3.0.0
7fb601b2c000-7fb601d2b000 ---p 0001c000 fd:03 105384823
/usr/lib64/libsasl2.so.3.0.0
7fb601d2b000-7fb601d2c000 r--p 0001b000 fd:03 105384823
/usr/lib64/libsasl2.so.3.0.0
7fb601d2c000-7fb601d2d000 rw-p 0001c000 fd:03 105384823
/usr/lib64/libsasl2.so.3.0.0
7fb601d2d000-7fb601d3b000 r-xp 00000000 fd:03 105385749
/usr/lib64/liblber-2.4.so.2.10.3
7fb601d3b000-7fb601f3a000 ---p 0000e000 fd:03 105385749
/usr/lib64/liblber-2.4.so.2.10.3
7fb601f3a000-7fb601f3b000 r--p 0000d000 fd:03 105385749
/usr/lib64/liblber-2.4.so.2.10.3
7fb601f3b000-7fb601f3c000 rw-p 0000e000 fd:03 105385749
/usr/lib64/liblber-2.4.so.2.10.3
7fb601f3c000-7fb601f3f000 r-xp 00000000 fd:03 105384892
/usr/lib64/libkeyutils.so.1.5
7fb601f3f000-7fb60213e000 ---p 00003000 fd:03 105384892
/usr/lib64/libkeyutils.so.1.5
7fb60213e000-7fb60213f000 r--p 00002000 fd:03 105384892
/usr/lib64/libkeyutils.so.1.5
7fb60213f000-7fb602140000 rw-p 00003000 fd:03 105384892
/usr/lib64/libkeyutils.so.1.5
7fb602140000-7fb60214d000 r-xp 00000000 fd:03 105385992
/usr/lib64/libkrb5support.so.0.1
7fb60214d000-7fb60234d000 ---p 0000d000 fd:03 105385992
/usr/lib64/libkrb5support.so.0.1
7fb60234d000-7fb60234e000 r--p 0000d000 fd:03 105385992
/usr/lib64/libkrb5support.so.0.1
7fb60234e000-7fb60234f000 rw-p 0000e000 fd:03 105385992
/usr/lib64/libkrb5support.so.0.1
7fb60234f000-7fb602364000 r-xp 00000000 fd:03 105386122
/usr/lib64/libz.so.1.2.7
7fb602364000-7fb602563000 ---p 00015000 fd:03 105386122
/usr/lib64/libz.so.1.2.7
7fb602563000-7fb602564000 r--p 00014000 fd:03 105386122
/usr/lib64/libz.so.1.2.7
7fb602564000-7fb602565000 rw-p 00015000 fd:03 105386122
/usr/lib64/libz.so.1.2.7
7fb602565000-7fb602594000 r-xp 00000000 fd:03 105385823
/usr/lib64/libk5crypto.so.3.1
7fb602594000-7fb602793000 ---p 0002f000 fd:03 105385823
/usr/lib64/libk5crypto.so.3.1
7fb602793000-7fb602795000 r--p 0002e000 fd:03 105385823
/usr/lib64/libk5crypto.so.3.1
7fb602795000-7fb602796000 rw-p 00030000 fd:03 105385823
/usr/lib64/libk5crypto.so.3.1
7fb602796000-7fb602797000 rw-p 00000000 00:00 0
7fb602797000-7fb60279a000 r-xp 00000000 fd:03 105384803
/usr/lib64/libcom_err.so.2.1
7fb60279a000-7fb602999000 ---p 00003000 fd:03 105384803
/usr/lib64/libcom_err.so.2.1
7fb602999000-7fb60299a000 r--p 00002000 fd:03 105384803
/usr/lib64/libcom_err.so.2.1
7fb60299a000-7fb60299b000 rw-p 00003000 fd:03 105384803
/usr/lib64/libcom_err.so.2.1
7fb60299b000-7fb602a71000 r-xp 00000000 fd:03 105385936
/usr/lib64/libkrb5.so.3.3
7fb602a71000-7fb602c71000 ---p 000d6000 fd:03 105385936
/usr/lib64/libkrb5.so.3.3
7fb602c71000-7fb602c7f000 r--p 000d6000 fd:03 105385936
/usr/lib64/libkrb5.so.3.3
7fb602c7f000-7fb602c82000 rw-p 000e4000 fd:03 105385936
/usr/lib64/libkrb5.so.3.3
7fb602c82000-7fb602cd8000 r-xp 00000000 fd:03 105384953
/usr/lib64/libldap_r-2.4.so.2.10.3
7fb602cd8000-7fb602ed8000 ---p 00056000 fd:03 105384953
/usr/lib64/libldap_r-2.4.so.2.10.3
7fb602ed8000-7fb602edb000 r--p 00056000 fd:03 105384953
/usr/lib64/libldap_r-2.4.so.2.10.3
7fb602edb000-7fb602edc000 rw-p 00059000 fd:03 105384953
/usr/lib64/libldap_r-2.4.so.2.10.3
7fb602edc000-7fb602ede000 rw-p 00000000 00:00 0
7fb602ede000-7fb602f29000 r-xp 00000000 fd:03 105385925
/usr/lib64/libgssapi_krb5.so.2.2
7fb602f29000-7fb603129000 ---p 0004b000 fd:03 105385925
/usr/lib64/libgssapi_krb5.so.2.2
7fb603129000-7fb60312a000 r--p 0004b000 fd:03 105385925
/usr/lib64/libgssapi_krb5.so.2.2
7fb60312a000-7fb60312c000 rw-p 0004c000 fd:03 105385925
/usr/lib64/libgssapi_krb5.so.2.2
7fb60312c000-7fb603360000 r-xp 00000000 fd:03 105390277
/usr/lib64/libcrypto.so.1.0.2k
7fb603360000-7fb603560000 ---p 00234000 fd:03 105390277
/usr/lib64/libcrypto.so.1.0.2k
7fb603560000-7fb60357c000 r--p 00234000 fd:03 105390277
/usr/lib64/libcrypto.so.1.0.2k
7fb60357c000-7fb603589000 rw-p 00250000 fd:03 105390277
/usr/lib64/libcrypto.so.1.0.2k
7fb603589000-7fb60358d000 rw-p 00000000 00:00 0
7fb60358d000-7fb6035f4000 r-xp 00000000 fd:03 105390278
/usr/lib64/libssl.so.1.0.2k
7fb6035f4000-7fb6037f4000 ---p 00067000 fd:03 105390278
/usr/lib64/libssl.so.1.0.2k
7fb6037f4000-7fb6037f8000 r--p 00067000 fd:03 105390278
/usr/lib64/libssl.so.1.0.2k
7fb6037f8000-7fb6037ff000 rw-p 0006b000 fd:03 105390278
/usr/lib64/libssl.so.1.0.2k
7fb6037ff000-7fb603844000 r-xp 00000000 fd:03 104338700
/usr/pgsql-10/lib/libpq.so.5.10
7fb603844000-7fb603a43000 ---p 00045000 fd:03 104338700
/usr/pgsql-10/lib/libpq.so.5.10
7fb603a43000-7fb603a46000 r--p 00044000 fd:03 104338700
/usr/pgsql-10/lib/libpq.so.5.10
7fb603a46000-7fb603a47000 rw-p 00047000 fd:03 104338700
/usr/pgsql-10/lib/libpq.so.5.10
7fb603a47000-7fb603a7c000 r-xp 00000000 fd:03 106174474
/usr/lib64/perl5/vendor_perl/auto/DBD/Pg/Pg.so
7fb603a7c000-7fb603c7c000 ---p 00035000 fd:03 106174474
/usr/lib64/perl5/vendor_perl/auto/DBD/Pg/Pg.so
7fb603c7c000-7fb603c7d000 r--p 00035000 fd:03 106174474
/usr/lib64/perl5/vendor_perl/auto/DBD/Pg/Pg.so
7fb603c7d000-7fb603c80000 rw-p 00036000 fd:03 106174474
/usr/lib64/perl5/vendor_perl/auto/DBD/Pg/Pg.so
7fb603c80000-7fb603c87000 r-xp 00000000 fd:03 106174465
/usr/lib64/perl5/vendor_perl/auto/version/vxs/vxs.so
7fb603c87000-7fb603e86000 ---p 00007000 fd:03 106174465
/usr/lib64/perl5/vendor_perl/auto/version/vxs/vxs.so
7fb603e86000-7fb603e87000 r--p 00006000 fd:03 106174465
/usr/lib64/perl5/vendor_perl/auto/version/vxs/vxs.so
7fb603e87000-7fb603e88000 rw-p 00007000 fd:03 106174465
/usr/lib64/perl5/vendor_perl/auto/version/vxs/vxs.so
7fb603e88000-7fb603e8a000 r-xp 00000000 fd:03 105385574
/usr/lib64/perl5/auto/I18N/Langinfo/Langinfo.so
7fb603e8a000-7fb60408a000 ---p 00002000 fd:03 105385574
/usr/lib64/perl5/auto/I18N/Langinfo/Langinfo.so
7fb60408a000-7fb60408b000 r--p 00002000 fd:03 105385574
/usr/lib64/perl5/auto/I18N/Langinfo/Langinfo.so
7fb60408b000-7fb60408c000 rw-p 00003000 fd:03 105385574
/usr/lib64/perl5/auto/I18N/Langinfo/Langinfo.so
7fb60408c000-7fb604094000 r-xp 00000000 fd:03 105385437
/usr/lib64/perl5/vendor_perl/auto/Data/Dumper/Dumper.so
7fb604094000-7fb604293000 ---p 00008000 fd:03 105385437
/usr/lib64/perl5/vendor_perl/auto/Data/Dumper/Dumper.so
7fb604293000-7fb604294000 r--p 00007000 fd:03 105385437
/usr/lib64/perl5/vendor_perl/auto/Data/Dumper/Dumper.so
7fb604294000-7fb604295000 rw-p 00008000 fd:03 105385437
/usr/lib64/perl5/vendor_perl/auto/Data/Dumper/Dumper.so
7fb604295000-7fb60429d000 r-xp 00000000 fd:03 105385325
/usr/lib64/perl5/vendor_perl/auto/Encode/Encode.so
7fb60429d000-7fb60449c000 ---p 00008000 fd:03 105385325
/usr/lib64/perl5/vendor_perl/auto/Encode/Encode.so
7fb60449c000-7fb60449d000 r--p 00007000 fd:03 105385325
/usr/lib64/perl5/vendor_perl/auto/Encode/Encode.so
7fb60449d000-7fb60449e000 rw-p 00008000 fd:03 105385325
/usr/lib64/perl5/vendor_perl/auto/Encode/Encode.so
7fb60449e000-7fb6044b0000 r-xp 00000000 fd:03 105385563
/usr/lib64/perl5/auto/POSIX/POSIX.so
7fb6044b0000-7fb6046af000 ---p 00012000 fd:03 105385563
/usr/lib64/perl5/auto/POSIX/POSIX.so
7fb6046af000-7fb6046b2000 r--p 00011000 fd:03 105385563
/usr/lib64/perl5/auto/POSIX/POSIX.so
7fb6046b2000-7fb6046b3000 rw-p 00014000 fd:03 105385563
/usr/lib64/perl5/auto/POSIX/POSIX.so
7fb6046b3000-7fb6046b6000 r-xp 00000000 fd:03 105385629
/usr/lib64/perl5/auto/Fcntl/Fcntl.so
7fb6046b6000-7fb6048b6000 ---p 00003000 fd:03 105385629
/usr/lib64/perl5/auto/Fcntl/Fcntl.so
7fb6048b6000-7fb6048b7000 r--p 00003000 fd:03 105385629
/usr/lib64/perl5/auto/Fcntl/Fcntl.so
7fb6048b7000-7fb6048b8000 rw-p 00004000 fd:03 105385629
/usr/lib64/perl5/auto/Fcntl/Fcntl.so
7fb6048b8000-7fb6048bf000 r-xp 00000000 fd:03 105385831
/usr/lib64/librt-2.17.so
7fb6048bf000-7fb604abe000 ---p 00007000 fd:03 105385831
/usr/lib64/librt-2.17.so
7fb604abe000-7fb604abf000 r--p 00006000 fd:03 105385831
/usr/lib64/librt-2.17.so
7fb604abf000-7fb604ac0000 rw-p 00007000 fd:03 105385831
/usr/lib64/librt-2.17.so
7fb604ac0000-7fb604ac5000 r-xp 00000000 fd:03 105385424
/usr/lib64/perl5/vendor_perl/auto/Time/HiRes/HiRes.so
7fb604ac5000-7fb604cc4000 ---p 00005000 fd:03 105385424
/usr/lib64/perl5/vendor_perl/auto/Time/HiRes/HiRes.so
7fb604cc4000-7fb604cc5000 r--p 00004000 fd:03 105385424
/usr/lib64/perl5/vendor_perl/auto/Time/HiRes/HiRes.so
7fb604cc5000-7fb604cc6000 rw-p 00005000 fd:03 105385424
/usr/lib64/perl5/vendor_perl/auto/Time/HiRes/HiRes.so
7fb604cc6000-7fb604cca000 r-xp 00000000 fd:03 105385607
/usr/lib64/perl5/auto/IO/IO.so
7fb604cca000-7fb604ec9000 ---p 00004000 fd:03 105385607
/usr/lib64/perl5/auto/IO/IO.so
7fb604ec9000-7fb604eca000 r--p 00003000 fd:03 105385607
/usr/lib64/perl5/auto/IO/IO.so
7fb604eca000-7fb604ecb000 rw-p 00004000 fd:03 105385607
/usr/lib64/perl5/auto/IO/IO.so
7fb604ecb000-7fb604ed6000 r-xp 00000000 fd:03 107610444
/usr/local/lib64/perl5/auto/List/Util/Util.so
7fb604ed6000-7fb6050d5000 ---p 0000b000 fd:03 107610444
/usr/local/lib64/perl5/auto/List/Util/Util.so
7fb6050d5000-7fb6050d6000 r--p 0000a000 fd:03 107610444
/usr/local/lib64/perl5/auto/List/Util/Util.so
7fb6050d6000-7fb6050d7000 rw-p 0000b000 fd:03 107610444
/usr/local/lib64/perl5/auto/List/Util/Util.so
7fb6050d7000-7fb6050f5000 r-xp 00000000 fd:03 106174312
/usr/lib64/perl5/vendor_perl/auto/DBI/DBI.so
7fb6050f5000-7fb6052f4000 ---p 0001e000 fd:03 106174312
/usr/lib64/perl5/vendor_perl/auto/DBI/DBI.so
7fb6052f4000-7fb6052f5000 r--p 0001d000 fd:03 106174312
/usr/lib64/perl5/vendor_perl/auto/DBI/DBI.so
7fb6052f5000-7fb6052f6000 rw-p 0001e000 fd:03 106174312
/usr/lib64/perl5/vendor_perl/auto/DBI/DBI.so
7fb6052f6000-7fb60b81f000 r--p 00000000 fd:03 105254492
/usr/lib/locale/locale-archive
7fb60b81f000-7fb60b821000 r-xp 00000000 fd:03 105383287
/usr/lib64/libfreebl3.so
7fb60b821000-7fb60ba20000 ---p 00002000 fd:03 105383287
/usr/lib64/libfreebl3.so
7fb60ba20000-7fb60ba21000 r--p 00001000 fd:03 105383287
/usr/lib64/libfreebl3.so
7fb60ba21000-7fb60ba22000 rw-p 00002000 fd:03 105383287
/usr/lib64/libfreebl3.so
7fb60ba22000-7fb60bbd8000 r-xp 00000000 fd:03 105385929
/usr/lib64/libc-2.17.so
7fb60bbd8000-7fb60bdd8000 ---p 001b6000 fd:03 105385929
/usr/lib64/libc-2.17.so
7fb60bdd8000-7fb60bddc000 r--p 001b6000 fd:03 105385929
/usr/lib64/libc-2.17.so
7fb60bddc000-7fb60bdde000 rw-p 001ba000 fd:03 105385929
/usr/lib64/libc-2.17.so
7fb60bdde000-7fb60bde3000 rw-p 00000000 00:00 0
7fb60bde3000-7fb60bdfa000 r-xp 00000000 fd:03 105387534
/usr/lib64/libpthread-2.17.so
7fb60bdfa000-7fb60bff9000 ---p 00017000 fd:03 105387534
/usr/lib64/libpthread-2.17.so
7fb60bff9000-7fb60bffa000 r--p 00016000 fd:03 105387534
/usr/lib64/libpthread-2.17.so
7fb60bffa000-7fb60bffb000 rw-p 00017000 fd:03 105387534
/usr/lib64/libpthread-2.17.so
7fb60bffb000-7fb60bfff000 rw-p 00000000 00:00 0
7fb60bfff000-7fb60c001000 r-xp 00000000 fd:03 105384924
/usr/lib64/libutil-2.17.so
7fb60c001000-7fb60c200000 ---p 00002000 fd:03 105384924
/usr/lib64/libutil-2.17.so
7fb60c200000-7fb60c201000 r--p 00001000 fd:03 105384924
/usr/lib64/libutil-2.17.so
7fb60c201000-7fb60c202000 rw-p 00002000 fd:03 105384924
/usr/lib64/libutil-2.17.so
7fb60c202000-7fb60c20a000 r-xp 00000000 fd:03 105384834
/usr/lib64/libcrypt-2.17.so
7fb60c20a000-7fb60c409000 ---p 00008000 fd:03 105384834
/usr/lib64/libcrypt-2.17.so
7fb60c409000-7fb60c40a000 r--p 00007000 fd:03 105384834
/usr/lib64/libcrypt-2.17.so
7fb60c40a000-7fb60c40b000 rw-p 00008000 fd:03 105384834
/usr/lib64/libcrypt-2.17.so
7fb60c40b000-7fb60c439000 rw-p 00000000 00:00 0
7fb60c439000-7fb60c539000 r-xp 00000000 fd:03 105386084
/usr/lib64/libm-2.17.so
7fb60c539000-7fb60c739000 ---p 00100000 fd:03 105386084
/usr/lib64/libm-2.17.so
7fb60c739000-7fb60c73a000 r--p 00100000 fd:03 105386084
/usr/lib64/libm-2.17.so
7fb60c73a000-7fb60c73b000 rw-p 00101000 fd:03 105386084
/usr/lib64/libm-2.17.so
7fb60c73b000-7fb60c73d000 r-xp 00000000 fd:03 105384839
/usr/lib64/libdl-2.17.so
7fb60c73d000-7fb60c93d000 ---p 00002000 fd:03 105384839
/usr/lib64/libdl-2.17.so
7fb60c93d000-7fb60c93e000 r--p 00002000 fd:03 105384839
/usr/lib64/libdl-2.17.so
7fb60c93e000-7fb60c93f000 rw-p 00003000 fd:03 105384839
/usr/lib64/libdl-2.17.so
7fb60c93f000-7fb60c955000 r-xp 00000000 fd:03 105385233
/usr/lib64/libnsl-2.17.so
7fb60c955000-7fb60cb54000 ---p 00016000 fd:03 105385233
/usr/lib64/libnsl-2.17.so
7fb60cb54000-7fb60cb55000 r--p 00015000 fd:03 105385233
/usr/lib64/libnsl-2.17.so
7fb60cb55000-7fb60cb56000 rw-p 00016000 fd:03 105385233
/usr/lib64/libnsl-2.17.so
7fb60cb56000-7fb60cb58000 rw-p 00000000 00:00 0
7fb60cb58000-7fb60cb6e000 r-xp 00000000 fd:03 105386078
/usr/lib64/libresolv-2.17.so
7fb60cb6e000-7fb60cd6e000 ---p 00016000 fd:03 105386078
/usr/lib64/libresolv-2.17.so
7fb60cd6e000-7fb60cd6f000 r--p 00016000 fd:03 105386078
/usr/lib64/libresolv-2.17.so
7fb60cd6f000-7fb60cd70000 rw-p 00017000 fd:03 105386078
/usr/lib64/libresolv-2.17.so
7fb60cd70000-7fb60cd72000 rw-p 00000000 00:00 0
7fb60cd72000-7fb60cef5000 r-xp 00000000 fd:03 105384687
/usr/lib64/perl5/CORE/libperl.so
7fb60cef5000-7fb60d0f5000 ---p 00183000 fd:03 105384687
/usr/lib64/perl5/CORE/libperl.so
7fb60d0f5000-7fb60d0f9000 r--p 00183000 fd:03 105384687
/usr/lib64/perl5/CORE/libperl.so
7fb60d0f9000-7fb60d0ff000 rw-p 00187000 fd:03 105384687
/usr/lib64/perl5/CORE/libperl.so
7fb60d0ff000-7fb60d100000 rw-p 00000000 00:00 0
7fb60d100000-7fb60d120000 r-xp 00000000 fd:03 105384942
/usr/lib64/ld-2.17.so
7fb60d122000-7fb60d223000 rw-p 00000000 00:00 0
7fb60d306000-7fb60d30d000 rw-p 00000000 00:00 0
7fb60d31d000-7fb60d31e000 rw-p 00000000 00:00 0
7fb60d31e000-7fb60d31f000 rw-p 00000000 00:00 0
7fb60d31f000-7fb60d320000 r--p 0001f000 fd:03 105384942
/usr/lib64/ld-2.17.so
7fb60d320000-7fb60d321000 rw-p 00020000 fd:03 105384942
/usr/lib64/ld-2.17.so
7fb60d321000-7fb60d322000 rw-p 00000000 00:00 0
7ffe9a5e4000-7ffe9a605000 rw-p 00000000 00:00 0
[stack]
7ffe9a7df000-7ffe9a7e1000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
(26657) [Tue Mar 31 09:24:36 2020] MCP Starting Bucardo version 5.6.0
(26657) [Tue Mar 31 09:24:36 2020] MCP Log level: normal
(26658) [Tue Mar 31 09:24:36 2020] MCP Master DB Local epoch:
1585661076.97289 DB epoch: 1585661076.97292
(26658) [Tue Mar 31 09:24:36 2020] MCP Master DB Local time: Tue Mar 31
09:24:36 2020 DB time: 2020-03-31 09:24:36.972925-04
(26658) [Tue Mar 31 09:24:36 2020] MCP Master DB Local timezone: EDT
(-0400) DB timezone: US/Eastern
(26658) [Tue Mar 31 09:24:36 2020] MCP Master DB Postgres version: 100012
(26658) [Tue Mar 31 09:24:36 2020] MCP Master DB Database port: 5432
(26658) [Tue Mar 31 09:24:36 2020] MCP Master DB Database host:
/var/run/postgresql
(26658) [Tue Mar 31 09:24:36 2020] MCP PID: 26658
(26658) [Tue Mar 31 09:24:36 2020] MCP Postgres library version: 90207
(26658) [Tue Mar 31 09:24:36 2020] MCP bucardo: /usr/sbin/bucardo
(26658) [Tue Mar 31 09:24:36 2020] MCP Bucardo.pm:
/usr/share/perl5/vendor_perl/Bucardo.pm
(26658) [Tue Mar 31 09:24:36 2020] MCP OS: linux Perl: /usr/bin/perl 5.16.3
(26658) [Tue Mar 31 09:24:36 2020] MCP DBI version: 1.627 DBD::Pg
version: 2.19.3 (21903) DBIx::Safe version: 1.2.5
(26658) [Tue Mar 31 09:24:36 2020] MCP Bucardo object:
batch => '0'
bcverbose => '1'
created => 'Tue Mar 31 09:24:36 2020'
dbdpgversion => '21903'
dbhlist => 'HASH(0xb0d9808)'
dbname => 'bucardo'
dbpass => '<not shown>'
dbuser => 'bucardo'
dryrun => '0'
exit_on_nosync => '0'
extraname => ''
listening => 'HASH(0x2036250)'
logclean => '0'
logcodes => 'HASH(0xb0b4aa8)'
logdest => ['/var/log/bucardo']
logextension => ''
logpid => '26658'
logprefix => 'MCP'
logseparate => '0'
masterdbh => 'DBI::db=HASH(0xb183798)'
mcp_backend => '26659'
mcp_clock_timestamp => 'clock_timestamp()'
mcppid => '26651'
pid_file => '/var/run/bucardo/bucardo.mcp.pid'
pidmap => 'HASH(0x2036298)'
quickstart => '0'
sendmail => '0'
sendmail_file => ''
sqlprefix => '/* Bucardo 5.6.0 */'
stop_file => '/var/run/bucardo/fullstopbucardo'
verbose => '1'
version => '5.6.0'
warning_file => ''
(26658) [Tue Mar 31 09:24:36 2020] MCP Bucardo config:
autosync_ddl => 'newcol'
bucardo_initial_version => '5.6.0'
bucardo_vac => '1'
bucardo_version => '5.6.0'
ctl_checkonkids_time => '10'
ctl_createkid_time => '0.5'
ctl_sleep => '0.2'
default_conflict_strategy => 'bucardo_latest'
default_email_from => '[email protected]'
default_email_host => 'localhost'
default_email_port => '25'
default_email_to => '[email protected]'
email_auth_pass => ''
email_auth_user => ''
email_debug_file => ''
endsync_sleep => '1.0'
flatfile_dir => '.'
host_safety_check => ''
isolation_level => 'repeatable read'
kid_deadlock_sleep => '0.5'
kid_nodeltarows_sleep => '0.5'
kid_pingtime => '60'
kid_restart_sleep => '1'
kid_serial_sleep => '0.5'
kid_sleep => '0.5'
log_conflict_file => 'bucardo_conflict.log'
log_level => 'normal'
log_level_number => '2'
log_microsecond => '0'
log_showlevel => '0'
log_showline => '0'
log_showpid => '1'
log_showsyncname => '1'
log_showtime => '3'
log_timer_format => ''
mcp_dbproblem_sleep => '15'
mcp_loop_sleep => '0.2'
mcp_pingtime => '60'
mcp_vactime => '60'
piddir => '/var/run/bucardo'
quick_delta_check => '1'
reason_file => 'bucardo.restart.reason.txt'
reload_config_timeout => '30'
semaphore_table => 'bucardo_status'
statement_chunk_size => '6000'
stats_script_url => 'http://www.bucardo.org/'
stopfile => 'fullstopbucardo'
syslog_facility => 'log_local1'
tcp_keepalives_count => '0'
tcp_keepalives_idle => '0'
tcp_keepalives_interval => '0'
vac_run => '30'
vac_sleep => '120'
warning_file => ''
(26658) [Tue Mar 31 09:24:37 2020] MCP Activating sync
"cargotel_common_sync"
(26658) [Tue Mar 31 09:24:37 2020] MCP Running validate_sync on
"cargotel_common_sync"
(26658) [Tue Mar 31 09:24:37 2020] MCP Connecting to database
"cargotel_cargotel" (source)
(26658) [Tue Mar 31 09:24:37 2020] MCP Database "cargotel_cargotel"
Local epoch: 1585661077.48551 DB epoch: 1585661077.48553
(26658) [Tue Mar 31 09:24:37 2020] MCP Database "cargotel_cargotel"
Local time: Tue Mar 31 09:24:37 2020 DB time: 2020-03-31 09:24:37.485529-04
(26658) [Tue Mar 31 09:24:37 2020] MCP Database "cargotel_cargotel"
Local timezone: EDT (-0400) DB timezone: US/Eastern
(26658) [Tue Mar 31 09:24:37 2020] MCP Database "cargotel_cargotel"
Postgres version: 100012
(26658) [Tue Mar 31 09:24:37 2020] MCP Database "cargotel_cargotel"
Database port: 5432
(26658) [Tue Mar 31 09:24:37 2020] MCP Database "cargotel_cargotel"
Database host: dbp3.cargotel.com
(26658) [Tue Mar 31 09:24:37 2020] MCP Connecting to database
"metro_metro_sync_prod" (source)
(26658) [Tue Mar 31 09:24:38 2020] MCP Database "metro_metro_sync_prod"
Local epoch: 1585661077.90004 DB epoch: 1585661077.92055
(26658) [Tue Mar 31 09:24:38 2020] MCP Database "metro_metro_sync_prod"
Local time: Tue Mar 31 09:24:37 2020 DB time: 2020-03-31 08:24:37.92055-05
(26658) [Tue Mar 31 09:24:38 2020] MCP Database "metro_metro_sync_prod"
Local timezone: EDT (-0400) DB timezone: US/Central
(26658) [Tue Mar 31 09:24:38 2020] MCP Database "metro_metro_sync_prod"
Postgres version: 110005
(26658) [Tue Mar 31 09:24:38 2020] MCP Database "metro_metro_sync_prod"
Database port: 5432
(26658) [Tue Mar 31 09:24:38 2020] MCP Database "metro_metro_sync_prod"
Database host: pgbouncer
(26658) [Tue Mar 31 09:24:45 2020] MCP Inspecting source table
"cargotel_common.load_classes" on database "cargotel_cargotel"
(26658) [Tue Mar 31 09:24:46 2020] MCP Inspecting source table
"cargotel_common.ref_msa_code" on database "cargotel_cargotel"
(26658) [Tue Mar 31 09:24:46 2020] MCP Inspecting source table
"cargotel_common.zip_loc" on database "cargotel_cargotel"
(26658) [Tue Mar 31 09:24:46 2020] MCP Activating sync "load_sync"
(26658) [Tue Mar 31 09:24:46 2020] MCP Running validate_sync on "load_sync"
<snip to save space>
(26658) [Tue Mar 31 09:24:57 2020] MCP Table metro.client_profile is
referenced by metro.zone_zip(zone_zip_parent_id_client_profile_id_fk),
which is not part of this sync!
(26658) [Tue Mar 31 09:24:57 2020] MCP Active syncs: 2
(26658) [Tue Mar 31 09:24:57 2020] MCP Entering main loop
(26658) [Tue Mar 31 09:24:57 2020] MCP Created VAC 26974
(26974) [Tue Mar 31 09:24:57 2020] VAC New VAC daemon. PID=26974
(26974) [Tue Mar 31 09:24:57 2020] VAC Connected to database
"cargotel_cargotel" with backend PID of 26978
(26979) [Tue Mar 31 09:24:58 2020] CTL New controller for sync
"cargotel_common_sync". Relgroup is "cargotel_common", dbs is
"metro_dbs". PID=26979
(26979) [Tue Mar 31 09:24:58 2020] CTL stayalive: 1 checksecs: 0 kicked: 1
(26979) [Tue Mar 31 09:24:58 2020] CTL kidsalive: 1 onetimecopy: 0
lifetimesecs: 0 (NULL) maxkicks: 0
(26979) [Tue Mar 31 09:24:58 2020] CTL Database "cargotel_cargotel"
backend PID: 26983
(26658) [Tue Mar 31 09:24:58 2020] MCP Created controller 26979 for sync
"cargotel_common_sync". Kick is 1
(26984) [Tue Mar 31 09:24:58 2020] CTL New controller for sync
"load_sync". Relgroup is "load", dbs is "metro_dbs". PID=26984
(26984) [Tue Mar 31 09:24:58 2020] CTL stayalive: 1 checksecs: 0 kicked: 1
(26984) [Tue Mar 31 09:24:58 2020] CTL kidsalive: 1 onetimecopy: 0
lifetimesecs: 0 (NULL) maxkicks: 0
(26984) [Tue Mar 31 09:24:58 2020] CTL Database "cargotel_cargotel"
backend PID: 26986
(26658) [Tue Mar 31 09:24:58 2020] MCP Created controller 26984 for sync
"load_sync". Kick is 1
(26974) [Tue Mar 31 09:24:58 2020] VAC Connected to database
"metro_metro_sync_prod" with backend PID of 698
(26979) [Tue Mar 31 09:24:58 2020] CTL Database "metro_metro_sync_prod"
backend PID: 697
(26988) [Tue Mar 31 09:24:58 2020] KID (cargotel_common_sync) New kid,
sync "cargotel_common_sync" alive=1 Parent=26979 PID=26988 kicked=1
(26984) [Tue Mar 31 09:24:58 2020] CTL Database "metro_metro_sync_prod"
backend PID: 700
(26991) [Tue Mar 31 09:24:58 2020] KID (load_sync) New kid, sync
"load_sync" alive=1 Parent=26984 PID=26991 kicked=1
bucardo status shows the following--other than the times nothing else
changes. logs on the receiving side show pings
[postgres@dbp3 ~]$ bucardo status
PID of Bucardo MCP: 26658
Name State Last
good Time Last I/D Last bad Time
======================+================================================+=======================+=============+===========+===========+=========
cargotel_common_sync | Good
| 09:24:59 | 10m 40s | 0/0 | none |
load_sync | Deleting based on cargotel_cargotel.metro.load
| Mar 30, 2020 20:18:10 | 13h 17m 29s | 51/123 | 09:24:29 | 11m 10s
[postgres@dbp3 ~]$
logs on the receiving side show really no activity:
[sudo] password for cargotel:
2020-03-31 13:37:58.147 UTC [4359] bucardo@metro_sync_prod LOG:
duration: 0.103 ms statement: SELECT 'DBD::Pg ping test'
2020-03-31 13:37:59.135 UTC [768] bucardo@metro_sync_prod LOG: duration:
0.113 ms statement: SELECT 'DBD::Pg ping test'
2020-03-31 13:38:58.087 UTC [4359] bucardo@metro_sync_prod LOG:
duration: 0.099 ms statement: SELECT 'DBD::Pg ping test'
2020-03-31 13:38:59.208 UTC [768] bucardo@metro_sync_prod LOG: duration:
0.120 ms statement: SELECT 'DBD::Pg ping test'
2020-03-31 13:39:00.468 UTC [698] bucardo@metro_sync_prod LOG: duration:
0.078 ms statement: DEALLOCATE dbdpg_p26974_9
2020-03-31 13:39:00.516 UTC [698] bucardo@metro_sync_prod LOG: duration:
0.029 ms statement: begin
2020-03-31 13:39:00.564 UTC [698] bucardo@metro_sync_prod LOG: duration:
0.067 ms parse dbdpg_p26974_10: SELECT bucardo.bucardo_purge_delta('45
seconds')
2020-03-31 13:39:00.612 UTC [698] bucardo@metro_sync_prod LOG: duration:
0.045 ms bind dbdpg_p26974_10: SELECT bucardo.bucardo_purge_delta('45
seconds')
2020-03-31 13:39:00.634 UTC [698] bucardo@metro_sync_prod LOG: duration:
21.780 ms execute dbdpg_p26974_10: SELECT
bucardo.bucardo_purge_delta('45 seconds')
2020-03-31 13:39:00.683 UTC [698] bucardo@metro_sync_prod LOG: duration:
0.211 ms statement: commit
2020-03-31 13:39:58.032 UTC [4359] bucardo@metro_sync_prod LOG:
duration: 0.089 ms statement: SELECT 'DBD::Pg ping test'
2020-03-31 13:39:59.282 UTC [768] bucardo@metro_sync_prod LOG: duration:
0.099 ms statement: SELECT 'DBD::Pg ping test'
This is the second time in about a week bucardo has gone down. Last time
I ended up removing all of it and starting from scratch--hoping not to
have to do that again.
I'd appreciate any help or suggestions on how to get going again.
Jeff Ross
-- Jeff Ross [email protected]
--
The contents of this e-mail and any attachments are intended solely for the
use of the named addressee(s) and may contain confidential and/or
privileged information. Any unauthorized use, copying, disclosure, or
distribution of the contents of this e-mail is strictly prohibited by the
sender and may be unlawful. If you are not the intended recipient, please
notify the sender immediately and delete this e-mail.
_______________________________________________
Bucardo-general mailing list
[email protected]
https://bucardo.org/mailman/listinfo/bucardo-general
