I wound up changing wsrep_sst_method to mariabackup instead of xtrabackup. Also, due to a bug (that seemed to be still present in 10.1.36) https://jira.mariadb.org/browse/MDEV-12956 I also had to configure datadir. Configured these in both /usr/local/pf/conf/mariadb/mariadb.conf.tt and /usr/local/pf/var/conf/mariadb.conf.
On Fri, Sep 28, 2018 at 2:57 PM Dave Barnum <[email protected]> wrote: > Here's the log from the second server.. > Sep 28 14:56:00 o-pf02-tdv mysqld: version = 4, > Sep 28 14:56:00 o-pf02-tdv mysqld: component = PRIMARY, > Sep 28 14:56:00 o-pf02-tdv mysqld: conf_id = 489, > Sep 28 14:56:00 o-pf02-tdv mysqld: members = 1/2 (joined/total), > Sep 28 14:56:00 o-pf02-tdv mysqld: act_id = 175, > Sep 28 14:56:00 o-pf02-tdv mysqld: last_appl. = -1, > Sep 28 14:56:00 o-pf02-tdv mysqld: protocols = 0/8/3 (gcs/repl/appl), > Sep 28 14:56:00 o-pf02-tdv mysqld: group UUID = > 8924558d-c343-11e8-af97-5e963e54a766 > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Note] WSREP: Flow-control interval: [23, 23] > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Note] WSREP: Trying to continue unpaused monitor > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 175) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: State transfer required: > Sep 28 14:56:00 o-pf02-tdv mysqld: Group state: > 8924558d-c343-11e8-af97-5e963e54a766:175 > Sep 28 14:56:00 o-pf02-tdv mysqld: Local state: > 00000000-0000-0000-0000-000000000000:-1 > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: New cluster view: global state: > 8924558d-c343-11e8-af97-5e963e54a766:175, view# 490: Primary, number of > nodes: 2, my index: 0, protocol version 3 > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Warning] WSREP: Gap in state sequence. Need state transfer. > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568327878400 > [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address > '10.18.0.31' --datadir '/var/lib/mysql/' --defaults-file > '/usr/local/pf/var/conf/mariadb.conf' --parent '63481' '' ''' > Sep 28 14:56:00 o-pf02-tdv mysqld: WSREP_SST: [INFO] Streaming with tar > (20180928 14:56:00.498) > Sep 28 14:56:00 o-pf02-tdv mysqld: WSREP_SST: [INFO] Using socat as > streamer (20180928 14:56:00.500) > Sep 28 14:56:00 o-pf02-tdv mysqld: WSREP_SST: [INFO] Evaluating socat -u > TCP-LISTEN:4444,reuseaddr stdio | tar xfi - --recursive-unlink -h; RC=( > ${PIPESTATUS[@]} ) (20180928 14:56:00.511) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: Prepared SST request: xtrabackup| > 10.18.0.31:4444/xtrabackup_sst > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: REPL Protocols: 8 (3, 2) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: Assign initial position for certification: 175, protocol > version: 3 > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568415561472 > [Note] WSREP: Service thread queue flushed. > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Warning] WSREP: Failed to prepare for incremental state transfer: Local > state UUID (00000000-0000-0000-0000-000000000000) does not match group > state UUID (8924558d-c343-11e8-af97-5e963e54a766): 1 (Operation not > permitted) > Sep 28 14:56:00 o-pf02-tdv mysqld: at > galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be > unavailable. > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Note] WSREP: Member 0.0 (o-pf02-tdv.domain.org) requested state transfer > from '*any*'. Selected 1.0 (o-pf01-tdv.domain.org)(SYNCED) as donor. > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 175) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: Requesting state transfer: success, donor: 1 > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663272192 > [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 > -> 8924558d-c343-11e8-af97-5e963e54a766:175 > Sep 28 14:56:00 o-pf02-tdv mysqld: tar: This does not look like a tar > archive > Sep 28 14:56:00 o-pf02-tdv mysqld: tar: Exiting with failure status due to > previous errors > Sep 28 14:56:00 o-pf02-tdv mysqld: WSREP_SST: [ERROR] Error while getting > data from donor node: exit codes: 0 2 (20180928 14:56:00.870) > Sep 28 14:56:00 o-pf02-tdv mysqld: WSREP_SST: [ERROR] Cleanup after exit > with status:32 (20180928 14:56:00.872) > Sep 28 14:56:00 o-pf02-tdv mysqld: WSREP_SST: [INFO] Removing the > sst_in_progress file (20180928 14:56:00.874) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568327878400 > [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role > 'joiner' --address '10.18.0.31' --datadir '/var/lib/mysql/' > --defaults-file '/usr/local/pf/var/conf/mariadb.conf' --parent '63481' > '' '': 32 (Broken pipe) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Warning] WSREP: 1.0 (o-pf01-tdv.domain.org): State transfer to 0.0 ( > o-pf02-tdv.domain.org) failed: -22 (Invalid argument) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568327878400 > [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from > joiner script. > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will > never receive state. Need to abort. > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Note] WSREP: gcomm: terminating thread > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568357234432 > [Note] WSREP: gcomm: joining thread > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663595264 > [ERROR] WSREP: SST failed: 32 (Broken pipe) > Sep 28 14:56:00 o-pf02-tdv mysqld: 2018-09-28 14:56:00 140568663595264 > [ERROR] Aborting > > > On Fri, Sep 28, 2018 at 2:50 PM Dave Barnum <[email protected]> wrote: > >> Hello. I'm trying to install a fresh PacketFence 8 cluster. I've gotten >> to the point where I join the second node. However, the `systemctl restart >> packetfence-mariadb` command doesn't return properly until it times out. >> On the first node, in the /usr/local/pf/logs/mariadb_error.log file, I'm >> seeing the following error: >> >> >> Sep 28 13:30:05 o-pf01-tdv mysqld: 2018-09-28 13:30:05 139646809925376 >> [Note] WSREP: Flow-control interval: [23, 23] >> Sep 28 13:30:05 o-pf01-tdv mysqld: 2018-09-28 13:30:05 139646809925376 >> [Note] WSREP: Trying to continue unpaused monitor >> Sep 28 13:30:05 o-pf01-tdv mysqld: 2018-09-28 13:30:05 139647107812096 >> [Note] WSREP: New cluster view: global state: >> 8924558d-c343-11e8-af97-5e963e54a766:167, view# 2: Primary, number of >> nodes: 2, my index: 1, protocol version 3 >> Sep 28 13:30:05 o-pf01-tdv mysqld: 2018-09-28 13:30:05 139647107812096 >> [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. >> Sep 28 13:30:05 o-pf01-tdv mysqld: 2018-09-28 13:30:05 139647107812096 >> [Note] WSREP: REPL Protocols: 8 (3, 2) >> Sep 28 13:30:05 o-pf01-tdv mysqld: 2018-09-28 13:30:05 139647107812096 >> [Note] WSREP: Assign initial position for certification: 167, protocol >> version: 3 >> Sep 28 13:30:05 o-pf01-tdv mysqld: 2018-09-28 13:30:05 139646866073344 >> [Note] WSREP: Service thread queue flushed. >> Sep 28 13:30:05 o-pf01-tdv mysqld_safe: A mysqld process already exists >> Sep 28 13:30:06 o-pf01-tdv mysqld_safe: A mysqld process already exists >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139646809925376 >> [Note] WSREP: Member 0.0 (o-pf02-tdv.domain.org) requested state >> transfer from '*any*'. Selected 1.0 (o-pf01-tdv.domain.org)(SYNCED) as >> donor. >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139646809925376 >> [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 167) >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139647107812096 >> [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139645723604736 >> [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'donor' --address ' >> 10.18.0.31:4444/xtrabackup_sst' --socket '/var/lib/mysql/mysql.sock' >> --datadir '/var/lib/mysql/' --defaults-file >> '/usr/local/pf/var/conf/mariadb.conf' '' --gtid >> '8924558d-c343-11e8-af97-5e963e54a766:167' --gtid-domain-id '0'' >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139647107812096 >> [Note] WSREP: sst_donor_thread signaled with 0 >> Sep 28 13:30:06 o-pf01-tdv mysqld: WSREP_SST: [INFO] Streaming with tar >> (20180928 13:30:06.298) >> Sep 28 13:30:06 o-pf01-tdv mysqld: WSREP_SST: [INFO] Using socat as >> streamer (20180928 13:30:06.304) >> Sep 28 13:30:06 o-pf01-tdv mysqld: WSREP_SST: [INFO] Streaming the backup >> to joiner at 10.18.0.31 4444 (20180928 13:30:06.320) >> Sep 28 13:30:06 o-pf01-tdv mysqld: WSREP_SST: [INFO] Evaluating >> innobackupex --defaults-file=/usr/local/pf/var/conf/mariadb.conf >> $INNOEXTRA --galera-info --stream=$sfmt ${TMPDIR} >> 2>${DATA}/innobackup.backup.log | socat -u stdio TCP:10.18.0.31:4444; >> RC=( ${PIPESTATUS[@]} ) (20180928 13:30:06.326) >> Sep 28 13:30:06 o-pf01-tdv mysqld: WSREP_SST: [ERROR] innobackupex >> finished with error: 1. Check /var/lib/mysql//innobackup.backup.log >> (20180928 13:30:06.343) >> Sep 28 13:30:06 o-pf01-tdv mysqld: WSREP_SST: [ERROR] Cleanup after exit >> with status:22 (20180928 13:30:06.349) >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139645723604736 >> [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup --role 'donor' >> --address '10.18.0.31:4444/xtrabackup_sst' --socket >> '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file >> '/usr/local/pf/var/conf/mariadb.conf' '' --gtid >> '8924558d-c343-11e8-af97-5e963e54a766:167' --gtid-domain-id '0' >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139645723604736 >> [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role >> 'donor' --address '10.18.0.31:4444/xtrabackup_sst' --socket >> '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file >> '/usr/local/pf/var/conf/mariadb.conf' '' --gtid >> '8924558d-c343-11e8-af97-5e963e54a766:167' --gtid-domain-id '0': 22 >> (Invalid argument) >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139645723604736 >> [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup --role 'donor' >> --address '10.18.0.31:4444/xtrabackup_sst' --socket >> '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file >> '/usr/local/pf/var/conf/mariadb.conf' '' --gtid >> '8924558d-c343-11e8-af97-5e963e54a766:167' --gtid-domain-id '0' >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139646809925376 >> [Warning] WSREP: 1.0 (o-pf01-tdv.wctnet.org): State transfer to 0.0 ( >> o-pf02-tdv.wctnet.org) failed: -22 (Invalid argument) >> Sep 28 13:30:06 o-pf01-tdv mysqld: 2018-09-28 13:30:06 139646809925376 >> [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 167) >> >> >> The contents of /var/lib/mysql//innobackup.backup.log are: >> IMPORTANT: Please check that the backup run completes successfully. >> At the end of a successful backup run innobackupex >> prints "completed OK!". >> >> xtrabackup: value 'xtrabackup_sst' may be wrong format for incremental >> option. >> >
_______________________________________________ PacketFence-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/packetfence-users
