Hi,
I am using a drbd device in an active/passive cluster setup with pacemaker. We
have dedicated connections for corosync heartbeats, drbd and a 10GB data
connection:
- A bonded 10GB network card for data traffic that will be accessed via a
virtual ip managed by pacemaker in 192.168.120.1/24. In the cluster nodes
MDA1PFP-S01 and MDA1PFP-S02 are assigned to 192.168.120.10 and 192.168.120.11.
- A dedicated back-to-back connection for corosync heartbeats in
192.168.121.1/24. MDA1PFP-PCS01 and MDA1PFP-S02 are assigned to 192.168.121.10
and 192.168.121.11. When the cluster is created, we use these as primary node
names and use the 10GB device as a second backup connection for increased
reliability: pcs cluster setup --name MDA1PFP MDA1PFP-PCS01,MDA1PFP-S01
MDA1PFP-PCS02,MDA1PFP-S02
- A dedicated back-to-back connection for drbd in 192.168.122.1/24. Hosts
MDA1PFP-DRBD01 and MDA1PFP-DRBD02 are assigned 192.168.23.10 and 192.168.123.11.
In my tests, I force a failover by
1. Shutdown the cluster node with the master with pcs cluster stop
2. Disable the network device for the virtual ip with ifdown and wait until
ping detects it
The initial state of the cluster is
MDA1PFP-S01 14:40:27 1803 0 ~ # pcs status
Cluster name: MDA1PFP
Last updated: Fri Sep 16 14:41:18 2016 Last change: Fri Sep 16 14:39:49
2016 by root via cibadmin on MDA1PFP-PCS01
Stack: corosync
Current DC: MDA1PFP-PCS02 (version 1.1.13-10.el7-44eb2dd) - partition with
quorum
2 nodes and 7 resources configured
Online: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
Full list of resources:
Master/Slave Set: drbd1_sync [drbd1]
Masters: [ MDA1PFP-PCS02 ]
Slaves: [ MDA1PFP-PCS01 ]
mda-ip (ocf::heartbeat:IPaddr2): Started MDA1PFP-PCS02
Clone Set: ping-clone [ping]
Started: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
ACTIVE (ocf::heartbeat:Dummy): Started MDA1PFP-PCS02
shared_fs (ocf::heartbeat:Filesystem): Started MDA1PFP-PCS02
PCSD Status:
MDA1PFP-PCS01: Online
MDA1PFP-PCS02: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
MDA1PFP-S01 14:41:19 1804 0 ~ # pcs resource --full
Master: drbd1_sync
Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1
notify=true
Resource: drbd1 (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=shared_fs
Operations: start interval=0s timeout=240 (drbd1-start-interval-0s)
promote interval=0s timeout=90 (drbd1-promote-interval-0s)
demote interval=0s timeout=90 (drbd1-demote-interval-0s)
stop interval=0s timeout=100 (drbd1-stop-interval-0s)
monitor interval=60s (drbd1-monitor-interval-60s)
Resource: mda-ip (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.120.20 cidr_netmask=32 nic=bond0
Operations: start interval=0s timeout=20s (mda-ip-start-interval-0s)
stop interval=0s timeout=20s (mda-ip-stop-interval-0s)
monitor interval=1s (mda-ip-monitor-interval-1s)
Clone: ping-clone
Resource: ping (class=ocf provider=pacemaker type=ping)
Attributes: dampen=5s multiplier=1000 host_list=pf-pep-dev-1 timeout=1
attempts=3
Operations: start interval=0s timeout=60 (ping-start-interval-0s)
stop interval=0s timeout=20 (ping-stop-interval-0s)
monitor interval=1 (ping-monitor-interval-1)
Resource: ACTIVE (class=ocf provider=heartbeat type=Dummy)
Operations: start interval=0s timeout=20 (ACTIVE-start-interval-0s)
stop interval=0s timeout=20 (ACTIVE-stop-interval-0s)
monitor interval=10 timeout=20 (ACTIVE-monitor-interval-10)
Resource: shared_fs (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/drbd1 directory=/shared_fs fstype=xfs
Operations: start interval=0s timeout=60 (shared_fs-start-interval-0s)
stop interval=0s timeout=60 (shared_fs-stop-interval-0s)
monitor interval=20 timeout=40 (shared_fs-monitor-interval-20)
MDA1PFP-S01 14:41:35 1805 0 ~ # pcs constraint --full
Location Constraints:
Resource: mda-ip
Enabled on: MDA1PFP-PCS01 (score:50) (id:location-mda-ip-MDA1PFP-PCS01-50)
Constraint: location-mda-ip
Rule: score=-INFINITY boolean-op=or (id:location-mda-ip-rule)
Expression: pingd lt 1 (id:location-mda-ip-rule-expr)
Expression: not_defined pingd (id:location-mda-ip-rule-expr-1)
Ordering Constraints:
start ping-clone then start mda-ip (kind:Optional)
(id:order-ping-clone-mda-ip-Optional)
promote drbd1_sync then start shared_fs (kind:Mandatory)
(id:order-drbd1_sync-shared_fs-mandatory)
Colocation Constraints:
ACTIVE with mda-ip (score:INFINITY) (id:colocation-ACTIVE-mda-ip-INFINITY)
drbd1_sync with mda-ip (score:INFINITY) (rsc-role:Master)
(with-rsc-role:Started) (id:colocation-drbd1_sync-mda-ip-INFINITY)
shared_fs with drbd1_sync (score:INFINITY) (rsc-role:Started)
(with-rsc-role:Master) (id:colocation-shared_fs-drbd1_sync-INFINITY)
The status after starting is:
Last updated: Fri Sep 16 14:39:57 2016 Last change: Fri Sep 16
14:39:49 2016 by root via cibadmin on MDA1PFP-PCS01
Stack: corosync
Current DC: MDA1PFP-PCS02 (version 1.1.13-10.el7-44eb2dd) - partition with
quorum
2 nodes and 7 resources configured
Online: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
Master/Slave Set: drbd1_sync [drbd1]
Masters: [ MDA1PFP-PCS02 ]
Slaves: [ MDA1PFP-PCS01 ]
mda-ip (ocf::heartbeat:IPaddr2): Started MDA1PFP-PCS02
Clone Set: ping-clone [ping]
Started: [ MDA1PFP-PCS01 MDA1PFP-PCS02 ]
ACTIVE (ocf::heartbeat:Dummy): Started MDA1PFP-PCS02
shared_fs (ocf::heartbeat:Filesystem): Started MDA1PFP-PCS02
The drbd is up as primary on MDA1PFP-PCS02 and mounted. Everything is fine.
If I do any of the two tests to force a failover the resources are moved to the
other node
but the drbd is not promoted to master on the new active node:
Last updated: Fri Sep 16 14:43:33 2016 Last change: Fri Sep 16
14:43:31 2016 by root via cibadmin on MDA1PFP-PCS01
Stack: corosync
Current DC: MDA1PFP-PCS01 (version 1.1.13-10.el7-44eb2dd) - partition with
quorum
2 nodes and 7 resources configured
Online: [ MDA1PFP-PCS01 ]
OFFLINE: [ MDA1PFP-PCS02 ]
Master/Slave Set: drbd1_sync [drbd1]
Slaves: [ MDA1PFP-PCS01 ]
mda-ip (ocf::heartbeat:IPaddr2): Started MDA1PFP-PCS01
Clone Set: ping-clone [ping]
Started: [ MDA1PFP-PCS01 ]
ACTIVE (ocf::heartbeat:Dummy): Started MDA1PFP-PCS01
I was able to trace this to the fencing in the drbd configuration
MDA1PFP-S01 14:41:44 1806 0 ~ # cat /etc/drbd.d/shared_fs.res
resource shared_fs {
disk /dev/mapper/rhel_mdaf--pf--pep--1-drbd;
disk {
fencing resource-only;
}
handlers {
fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
}
device /dev/drbd1;
meta-disk internal;
on MDA1PFP-S01 {
address 192.168.123.10:7789;
}
on MDA1PFP-S02 {
address 192.168.123.11:7789;
}
}
I am using drbd 8.4.7, drbd utils 8.9.5 and pacemaker 2.3.4-7.el7 with
corosyinc 0.9.143-15.el7 from the Centos7 repositories.
MDA1PFP-S01 15:00:20 1841 0 ~ # drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 5d50d9fb2a967d21c0f5746370ccc066d3a67f7d\ build\
by\ mockbuild@\,\ 2016-01-12\ 12:46:45
DRBDADM_API_VERSION=1
DRBD_KERNEL_VERSION_CODE=0x080407
DRBDADM_VERSION_CODE=0x080905
DRBDADM_VERSION=8.9.5
If I disable the fencing scripts everything works as expected. If enabled, no
node is promoted to master after failover. It seems to be a sticky modificaton
because once a failover is simulated with fencing scripts activated I cannot
get the cluster to work anymore. Even removing the setting from the DRBD
configuration does not help.
I discussed this on the Pacemaker mailing list and from a Pacemaker point of
view this should not happen. The nodes are still online so no fencing should
happen, however DRBD fences the wrong node?
Best wishes,
Jens Auer
--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.a...@cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
de.cgi.com/pflichtangaben.
CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
Group Inc. and its affiliates may be contained in this message. If you are not
a recipient indicated or intended in this message (or responsible for delivery
of this message to such person), or you think for any reason that this message
may have been addressed to you in error, you may not use or copy or deliver
this message to anyone else. In such case, you should destroy this message and
are asked to notify the sender by reply e-mail.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
