I've not done fencing in DRBD alone, so I am unable to offer specific
suggestions. I can speak to generally what you need though.
You can set DRBD's fencing policy to 'resource-and-stonith'. What this
does is tell DRBD "When you lose your peer, block IO and call a fence
against it". The fence action reaches out (usually via IPMI or managed
PDU) and forces the peer offline. After that, the surviving node will
proceed. This way, at no time will both nodes be operating in StandAlone
and Primary.
You will want to set a delay so that one of the nodes has a head start
when trying to fence the other. This way, in your test, when the
communication breaks but the nodes are still up, you remove the risk of
both nodes being fenced. What this does is say "when you want to fence
node 1, wait 15 seconds before doing so. when you want to fence node 2,
don't wait and immediately fence". Thus, when it's a break in
communications, you can predict which node will win the fence.
When a node really fails, it will obviously not try to fence, being
dead, so the healthy node will always win the fence and then take over.
How you actually fence the peer will depends on what options you have
available. Then you need a script that will actually do the work of
reaching out and killing the peer. As I mentioned, this is usually done
via IPMI (or branded out of band interfaces like iLO, DRAC, RSA, etc) or
by using managed PDUs, like the APC AP7900. To do this, you need to have
a scrip that reads certain environment variables set by DRBD, executes
the request and then returns an appropriate exit code based on success
or failure.
I wrote such a fence handler called "rhcs_fence" (based on
obliterate-peer.sh) which handles fencing by passing the request up to
rhcs. You should be able to fairly easily adapt it to work with your setup.
https://github.com/digimer/rhcs_fence
Hope this helps clarify things.
digimer
On 04/10/2013 01:22 PM, Shailesh Vaidya wrote:
Hi Don,
Yup 8.3.8 is quit old but need to work with it for now.
I am not using fencing and neither pacemaker or RHCS
What I observed is that after split-brain its getting disconnected and dropped
connection. both became unknown to each other. I am not sure is this issue with
my test procedure itself.
Do I need to make any additional configuration.
Thanks,
Shailesh Vaidya.
________________________________________
From: Digimer [li...@alteeve.ca]
Sent: Wednesday, April 10, 2013 8:11 PM
To: Shailesh Vaidya
Cc: Dan Barker; drbd-user@lists.linbit.com
Subject: Re: [DRBD-user] Not able to test Automatic split brain recovery
policies
To your immediate problem;
If you had configured fencing, drbd would not split-brain. Are you using
pacemaker or RHCS?
Secondly, 8.3.8 is very, very old. Upgrading to a newer 8.3.x version
would be a good idea.
Back to split-brain; DRBD declares a split-brain as soon as both nodes
are StandAlone and Primary. To recover, you need to tell DRBD which node
to consider "good" and then drop the changes on the peer and let the
good node sync to the other node.
On 04/10/2013 08:08 AM, Shailesh Vaidya wrote:
I have followed same procedure (disable Ethernet card) etc and after
that drbd status on both the nodes
[root@drbd1 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by
mockbu...@builder10.centos.org, 2010-06-04 08:04:09
0: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
ns:4 nr:0 dw:12 dr:82 al:1 bm:3 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:4
[root@drbd1 ~]#
[root@drbd2 ~]# cat /proc/drbd
version: 8.3.8 (api:88/proto:86-94)
GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by
mockbu...@builder10.centos.org, 2010-06-04 08:04:09
0: cs:StandAlone ro:Secondary/Unknown ds:UpToDate/DUnknown r----
ns:0 nr:4 dw:56 dr:42 al:1 bm:4 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:4
[root@drbd2 ~]#
/var/log/messages shows
Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
/sbin/drbdadm initial-split-brain minor-0
Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
/sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
Apr 10 07:51:35 localhost kernel: block drbd0: Split-Brain detected but
unresolved, dropping connection!
Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
/sbin/drbdadm split-brain minor-0
Apr 10 07:51:35 localhost kernel: block drbd0: helper command:
/sbin/drbdadm split-brain minor-0 exit code 0 (0x0)
Apr 10 07:51:35 localhost kernel: block drbd0: conn( WFReportParams ->
Disconnecting )
Apr 10 07:51:35 localhost kernel: block drbd0: error receiving
ReportState, l: 4!
Apr 10 07:51:35 localhost kernel: block drbd0: asender terminated
Apr 10 07:51:35 localhost kernel: block drbd0: Terminating asender thread
Apr 10 07:51:35 localhost kernel: block drbd0: Connection closed
Apr 10 07:51:35 localhost kernel: block drbd0: conn( Disconnecting ->
StandAlone )
Apr 10 07:51:35 localhost kernel: block drbd0: receiver terminated
Apr 10 07:51:35 localhost kernel: block drbd0: Terminating receiver thread
Now if I do ‘drbdadm connect r0’ on both the machines then,
Apr 10 07:56:37 localhost kernel: block drbd0: uuid_compare()=100 by rule 90
Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
/sbin/drbdadm initial-split-brain minor-0
Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
/sbin/drbdadm initial-split-brain minor-0 exit code 0 (0x0)
Apr 10 07:56:37 localhost kernel: block drbd0: Split-Brain detected, 1
primaries, automatically solved. Sync from peer node
Apr 10 07:56:37 localhost kernel: block drbd0: peer( Unknown -> Primary
) conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate )
Apr 10 07:56:37 localhost kernel: block drbd0: conn( WFBitMapT ->
WFSyncUUID )
Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
/sbin/drbdadm before-resync-target minor-0
Apr 10 07:56:37 localhost kernel: block drbd0: helper command:
/sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
Apr 10 07:56:37 localhost kernel: block drbd0: conn( WFSyncUUID ->
SyncTarget ) disk( UpToDate -> Inconsistent )
Apr 10 07:56:37 localhost kernel: block drbd0: Began resync as
SyncTarget (will sync 4 KB [1 bits set]).
Apr 10 07:56:37 localhost kernel: block drbd0: Resync done (total 1 sec;
paused 0 sec; 4 K/sec)
Regards,
Shailesh Vaidya
*From:*drbd-user-boun...@lists.linbit.com
[mailto:drbd-user-boun...@lists.linbit.com] *On Behalf Of *Dan Barker
*Sent:* Wednesday, April 10, 2013 5:16 PM
*To:* drbd-user@lists.linbit.com
*Subject:* Re: [DRBD-user] Not able to test Automatic split brain
recovery policies
You don’t show the status of the nodes, but I imagine you have two
primary nodes. There is no handler specified for two primary nodes. Did
you have two primary, disconnected nodes?
It shouldn’t be possible to create split brain without writing on both
nodes.
Dan
*From:*drbd-user-boun...@lists.linbit.com
<mailto:drbd-user-boun...@lists.linbit.com>
[mailto:drbd-user-boun...@lists.linbit.com] *On Behalf Of *Shailesh Vaidya
*Sent:* Wednesday, April 10, 2013 1:58 AM
*To:* drbd-user@lists.linbit.com <mailto:drbd-user@lists.linbit.com>
*Subject:* [DRBD-user] Not able to test Automatic split brain recovery
policies
Hello,
I am using DRBD 8.3.8
I have configured Automatic split brain recovery policies as below in
/etc/drbd.conf
net {
max-buffers 2048;
ko-count 4;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
}
My both machines are Virtual machines so not connected actual
back-to-back connection. To reproduce split-brain, I am using below
procedure,
1.On Primary disable Ethernet card from ‘Virtual Machine properties’
2.Wait to Secondery to start switch over and again enable Ethernet card
on Primary
Log shows mw that split-brain is occurred , however its shows connection
dropped.
Apr 9 10:30:15 drbd1 kernel: block drbd0: uuid_compare()=100 by rule 90
Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm
initial-split-brain minor-0
Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm
initial-split-brain minor-0 exit code 0 (0x0)
Apr 9 10:30:15 drbd1 kernel: block drbd0: Split-Brain detected but
unresolved, dropping connection!
Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm
split-brain minor-0
Apr 9 10:30:15 drbd1 kernel: block drbd0: helper command: /sbin/drbdadm
split-brain minor-0 exit code 0 (0x0)
Apr 9 10:30:15 drbd1 kernel: block drbd0: conn( WFReportParams ->
Disconnecting )
Full DRBD conf file
[root@drbd1 ~]# cat /etc/drbd.conf
global {
usage-count no;
}
resource r0 {
protocol C;
#incon-degr-cmd "echo !DRBD! pri on incon-degr | wall ; sleep 60 ; halt -f";
on drbd1 {
device /dev/drbd0;
disk /dev/sda3;
address 10.55.199.51:7789;
meta-disk internal;
}
on drbd2 {
device /dev/drbd0;
disk /dev/sda3;
address 10.55.199.52:7789;
meta-disk internal;
}
disk {
on-io-error detach;
}
net {
max-buffers 2048;
ko-count 4;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
}
syncer {
rate 25M;
al-extents 257; # must be a prime number
}
startup {
wfc-timeout 20;
degr-wfc-timeout 120; # 2 minutes.
}
}
[root@drbd1 ~]# vi /var/log/messages
[root@drbd1 ~]#
[root@drbd1 ~]# cat /etc/drbd.conf
global {
usage-count no;
}
resource r0 {
protocol C;
#incon-degr-cmd "echo !DRBD! pri on incon-degr | wall ; sleep 60 ; halt -f";
on drbd1 {
device /dev/drbd0;
disk /dev/sda3;
address 10.55.199.51:7789;
meta-disk internal;
}
on drbd2 {
device /dev/drbd0;
disk /dev/sda3;
address 10.55.199.52:7789;
meta-disk internal;
}
disk {
on-io-error detach;
}
net {
max-buffers 2048;
ko-count 4;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
}
syncer {
rate 25M;
al-extents 257; # must be a prime number
}
startup {
wfc-timeout 20;
degr-wfc-timeout 120; # 2 minutes.
}
}
[root@drbd1 ~]#
Is this configuration issue or my testing procedure is not proper?
Regards,
Shailesh Vaidya
DISCLAIMER ========== This e-mail may contain privileged and
confidential information which is the property of Persistent Systems
Ltd. It is intended only for the use of the individual or entity to
which it is addressed. If you are not the intended recipient, you are
not authorized to read, retain, copy, print, distribute or use this
message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems
Ltd. does not accept any liability for virus infected mails.
DISCLAIMER ========== This e-mail may contain privileged and
confidential information which is the property of Persistent Systems
Ltd. It is intended only for the use of the individual or entity to
which it is addressed. If you are not the intended recipient, you are
not authorized to read, retain, copy, print, distribute or use this
message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems
Ltd. does not accept any liability for virus infected mails.
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Persistent Systems Ltd. It is intended only for the use of the
individual or entity to which it is addressed. If you are not the intended
recipient, you are not authorized to read, retain, copy, print, distribute or
use this message. If you have received this communication in error, please
notify the sender and delete all copies of this message. Persistent Systems
Ltd. does not accept any liability for virus infected mails.
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
_______________________________________________
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user
