Hi
I have setup a master/standby on PostgreSQL95 on two test servers and trialing
out repmgr. (https://github.com/2ndQuadrant/repmgr/)
I am testing a switchover using the following:
-bash-4.1$ repmgr -f /etc/repmgr/9.5/repmgr.conf -C /etc/repmgr/9.5/repmgr.conf
standby switchover -L DEBUG -v
The switchover appears to hang at the last part of the switchover process....
NOTICE: restarting server using '/usr/pgsql-9.5/bin/pg_ctl -w -D
/var/lib/pgsql/9.5/data -m fast restart'
pg_ctl: PID file "/var/lib/pgsql/9.5/data/postmaster.pid" does not exist
Is server running?
starting server anyway
It appears to have worked though as when I run the cluster show command on both
servers it showing the switchover.
-bash-4.1$ repmgr -f /etc/repmgr/9.5/repmgr.conf cluster show
Role | Name | Upstream | Connection String
----------+----------------|----------------|-------------------------------------------
* master | itupl-postgen2 | | host=10.70.3.252 dbname=repmgr
user=repmgr
standby | itupl-postgen1 | itupl-postgen2 | host=10.70.3.251 dbname=repmgr
user=repmgr
It is also showing correctly in repl_nodes table of the two databases.
Why is it hanging?? Thank you for your help...
Here is the complete output:
-----------------------------------------------
-bash-4.1$ repmgr -f /etc/repmgr/9.5/repmgr.conf -C /etc/repmgr/9.5/repmgr.conf
standby switchover -L DEBUG -v
NOTICE: using configuration file "/etc/repmgr/9.5/repmgr.conf"
NOTICE: switching current node 2 to master server and demoting current master
to standby...
DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr
fallback_application_name='repmgr''
DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery()
INFO: retrieving node list for cluster 'repmgr_cluster'
DEBUG: get_master_connection():
SELECT id, conninfo, CASE WHEN type = 'master' THEN 1 ELSE 2 END AS
type_priority FROM "repmgr_repmgr_cluster".repl_nodes WHERE cluster =
'repmgr_cluster' AND type != 'witness' ORDER BY active DESC,
type_priority, priority, id
INFO: checking role of cluster node '1'
DEBUG: connecting to: 'host=10.70.3.251 dbname=repmgr user=repmgr
fallback_application_name='repmgr''
DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery()
DEBUG: get_master_connection(): current master node is 1
DEBUG: get_node_record():
SELECT id, type, upstream_node_id, name, conninfo, slot_name, priority,
active FROM "repmgr_repmgr_cluster".repl_nodes WHERE cluster =
'repmgr_cluster' AND id = 1
DEBUG: remote node name is "itupl-postgen1"
DEBUG: test_ssh_connection(): executing ssh -o Batchmode=yes 10.70.3.251
/bin/true 2>/dev/null
DEBUG: get_pg_setting(): SELECT name, setting FROM pg_catalog.pg_settings
WHERE name = 'data_directory'
DEBUG: get_pg_setting(): returned value is "/var/lib/pgsql/9.5/data"
DEBUG: master's data directory is: /var/lib/pgsql/9.5/data
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls
'/var/lib/pgsql/9.5/data/PG_VERSION' >/dev/null 2>&1 && echo 1 || echo 0
DEBUG: remote_command(): output returned was:
1
DEBUG: PG_VERSION found in /var/lib/pgsql/9.5/data
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls
'/usr/pgsql-9.5/bin/pg_rewind' >/dev/null 2>&1 && echo 1 || echo 0
DEBUG: remote_command(): output returned was:
1
DEBUG: guc_set():
SELECT true FROM pg_catalog.pg_settings WHERE name = 'full_page_writes' AND
setting = 'off'
DEBUG: guc_set():
SELECT true FROM pg_catalog.pg_settings WHERE name = 'wal_log_hints' AND
setting = 'on'
INFO: looking for file "/etc/repmgr/9.5/repmgr.conf" on remote server
"10.70.3.251"
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls
'/etc/repmgr/9.5/repmgr.conf' >/dev/null 2>&1 && echo 1 || echo 0
DEBUG: remote_command(): output returned was:
1
INFO: remote configuration file "/etc/repmgr/9.5/repmgr.conf" found on remote
server
DEBUG: remote_archive_config_dir: /tmp/repmgr-itupl-postgen1-archive
DEBUG: Executing:
/usr/pgsql-9.5/bin/repmgr standby archive-config -f
'/etc/repmgr/9.5/repmgr.conf'
--config-archive-dir='/tmp/repmgr-itupl-postgen1-archive'
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251
/usr/pgsql-9.5/bin/repmgr standby archive-config -f
'/etc/repmgr/9.5/repmgr.conf'
--config-archive-dir='/tmp/repmgr-itupl-postgen1-archive'
WARNING: nonstandard use of escape in a string literal
LINE 1: ...config_file, regexp_replace(config_file, '^.*\/',''...
^
HINT: Use the escape string syntax for escapes, e.g., E'\r\n'.
NOTICE: 3 files copied to /tmp/repmgr-itupl-postgen1-archive
DEBUG: remote_command(): output returned was:
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251
/usr/pgsql-9.5/bin/pg_ctl -D '/var/lib/pgsql/9.5/data' -m fast -W stop
>/dev/null 2>&1 && echo 1 || echo 0
DEBUG: remote_command(): output returned was:
1
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 ls
'/var/lib/pgsql/9.5/data/postmaster.pid' >/dev/null 2>&1 && echo 1 || echo 0
DEBUG: remote_command(): output returned was:
0
NOTICE: current master has been stopped
INFO: connecting to standby database
DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr
fallback_application_name='repmgr''
INFO: connected to standby, checking its state
DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery()
INFO: retrieving node list for cluster 'repmgr_cluster'
DEBUG: get_master_connection():
SELECT id, conninfo, CASE WHEN type = 'master' THEN 1 ELSE 2 END AS
type_priority FROM "repmgr_repmgr_cluster".repl_nodes WHERE cluster =
'repmgr_cluster' AND type != 'witness' ORDER BY active DESC,
type_priority, priority, id
INFO: checking role of cluster node '1'
DEBUG: connecting to: 'host=10.70.3.251 dbname=repmgr user=repmgr
fallback_application_name='repmgr''
ERROR: connection to database failed: could not connect to server: Connection
refused
Is the server running on host "10.70.3.251" and accepting
TCP/IP connections on port 5432?
INFO: checking role of cluster node '2'
DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr
fallback_application_name='repmgr''
DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery()
NOTICE: promoting standby
DEBUG: get_pg_setting(): SELECT name, setting FROM pg_catalog.pg_settings
WHERE name = 'data_directory'
DEBUG: get_pg_setting(): returned value is "/var/lib/pgsql/9.5/data"
NOTICE: promoting server using '/usr/pgsql-9.5/bin/pg_ctl -D
/var/lib/pgsql/9.5/data promote'
server promoting
INFO: reconnecting to promoted server
DEBUG: connecting to: 'host=10.70.3.252 dbname=repmgr user=repmgr
fallback_application_name='repmgr''
DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery()
DEBUG: is_standby(): SELECT pg_catalog.pg_is_in_recovery()
DEBUG: setting node 2 as master and marking existing master as failed
DEBUG: begin_transaction()
DEBUG: commit_transaction()
NOTICE: STANDBY PROMOTE successful
DEBUG: create_event_record():
INSERT INTO "repmgr_repmgr_cluster".repl_events ( node_id,
event, successful, details )
VALUES ($1, $2, $3, $4) RETURNING event_timestamp
DEBUG: create_event_record(): Event timestamp is "2017-05-22
16:56:06.860066+09:30"
NOTICE: Executing pg_rewind on old master server
DEBUG: pg_rewind command is:
'/usr/pgsql-9.5/bin/pg_rewind' -D '/var/lib/pgsql/9.5/data'
--source-server=\'host=10.70.3.252 dbname=repmgr user=repmgr\'
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251
'/usr/pgsql-9.5/bin/pg_rewind' -D '/var/lib/pgsql/9.5/data'
--source-server=\'host=10.70.3.252 dbname=repmgr user=repmgr\'
DEBUG: remote_command(): output returned was:
servers diverged at WAL position 1/1D000098 on timeline 11
no rewind required
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251
/usr/pgsql-9.5/bin/repmgr standby restore-config -D '/var/lib/pgsql/9.5/data'
--config-archive-dir='/tmp/repmgr-itupl-postgen1-archive'
ERROR: unable to determine cluster name - please provide a valid configuration
file with -c/--config-file
HINT: Use -F/--force to continue anyway
DEBUG: remote_command(): output returned was:
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251 test -e
'/var/lib/pgsql/9.5/data/recovery.done' && rm -f
'/var/lib/pgsql/9.5/data/recovery.done'
DEBUG: remote_command(): output returned was:
DEBUG: Executing:
/usr/pgsql-9.5/bin/repmgr -D '/var/lib/pgsql/9.5/data' -f
'/etc/repmgr/9.5/repmgr.conf' -h 10.70.3.252 -d repmgr -U repmgr standby follow
DEBUG: remote_command(): ssh -o Batchmode=yes 10.70.3.251
/usr/pgsql-9.5/bin/repmgr -D '/var/lib/pgsql/9.5/data' -f
'/etc/repmgr/9.5/repmgr.conf' -h 10.70.3.252 -d repmgr -U repmgr standby follow
NOTICE: restarting server using '/usr/pgsql-9.5/bin/pg_ctl -w -D
/var/lib/pgsql/9.5/data -m fast restart'
pg_ctl: PID file "/var/lib/pgsql/9.5/data/postmaster.pid" does not exist
Is server running?
starting server anyway
Regards
Dylan
