Re: [DRBD-user] drbd fencing policy problem, upgrading from 8.2.7 - 8.3.6
Lars,
That bitmap adjustment did the trick, the resource-only fencing
is working now. I'll investigate the new metadata tools, the less
of this stuff we have to maintain the better. Thanks!
Peter
-Original Message-
From: drbd-user-boun...@lists.linbit.com [mailto:drbd-user-
boun...@lists.linbit.com] On Behalf Of Lars Ellenberg
Sent: Saturday, February 06, 2010 1:12 PM
To: drbd-user@lists.linbit.com
Subject: Re: [DRBD-user] drbd fencing policy problem,upgrading from
8.2.7 - 8.3.6
On Fri, Feb 05, 2010 at 12:46:24PM -0500, Petrakis, Peter wrote:
Hi All,
We use resource-only fencing currently which is causing us a problem
when we're bringing up a cluster for the first time. This all worked
well with 8.2.7. We only have one node active at this time and the
fencing handler is bailing with '5' which is correct. The problem
is,
this return failure is stopping us from becoming Primary.
block drbd5: helper command: /usr/lib/spine/bin/avance_drbd_helper
fence-peer minor-5 exit code 5 (0x500)
block drbd5: State change failed: Refusing to be Primary without at
least one UpToDate disk
block drbd5: state = { cs:StandAlone ro:Secondary/Unknown
ds:Consistent/DUnknown r--- }
block drbd5: wanted = { cs:StandAlone ro:Primary/Unknown
ds:Consistent/DUnknown r--- }
I'm quoting our internal bugzilla here:
scenario:
A (Primary) --- b (Secondary)
A (Primary) b (down, crashed; does not know that it is out of
date)
a (down)
clean shutdown or crashed, does not matter, knows b is Outdated...
a (still down) b (reboot)
after init dead, heartbeat tries to promote
current behaviour:
b (Secondary, Consistent, pdsk DUnknown)
calls dopd, times out, considers other node as outdated,
and goes Primary.
proposal:
only try to outdate peer if local data is UpToDate.
- works for primary crash (secondary data is UpToDate)
- works for secondary crash (naturally)
- works for primary reboot while secondary offline,
because pdsk is marked Outdated in local meta data,
so we can become UpToDate on restart.
- additionally works for previously described scenario
because crashed secondary has pdsk Unknown,
thus comes up as Consistent (not UpToDate)
avoids one more way to create diverging data sets.
if you want to force Consistent to be UpToDate,
you'd then need either fiddle with drbdmeta,
or maybe just the good old --overwrite-data-of-peer does the trick?
does not work for cluster crash,
when only former secondary comes up.
--- Comment #1 From Florian Haas 2009-07-03 09:07:57 [reply]
--
-
Bumping severity to normal and setting target milestone to 8.3.3.
This is not an enhancement feature, it's a real (however subtle) bug.
This
tends to break dual-Primary setups with fencing, such as with GFS:
- Dual Primary configuration, GFS and CMAN running.
- Replication link goes away.
- Both nodes are now degraded clusters.
- GFS/CMAN initiates fencing, one node gets rebooted.
- Node reboots, link is still down. Since we were degraded clusters
to begin
with, degr-wfc-timeout now applies, which is finite by default.
- After the timeout expires, the recovered node is now Consistent and
attempts
to fence the peer, when it should not.
- Since the network link is still down, fencing fails, but we now
assume the
peer is dead, and the node becomes Primary anyway.
- We have split brain, diverging datasets, all our fencing precautions
are moot.
--- Comment #3 From Philipp Reisner 2009-08-25 14:42:58 [reply]
---
Proposed solution:
Just to recap, the expected exit codes of the fence-peer-hander:
3 Peer's disk state was already Inconsistent.
4 Peer's disk state was successfully set to Outdated (or was Outdated
to begin with).
5 Connection to the peer node failed, peer could not be reached.
6 Peer refused to be outdated because the affected resource was in
the
primary role.
7 Peer node was successfully fenced off the cluster. This should
never
occur
unless fencing is set to resource-and-stonith for the affected
resource.
Now, if we get a 5 (peer not reachable) and we are not UpToDate (that
means we
are only Consistent) then refuse to become primary and do not consider
the peer
as outdated.
The change in DRBD is minimal, but we need then to implement exit code
5 also
in crm-fence-peer.sh
--- Comment #4 From Philipp Reisner 2009-08-26 19:15:33 [reply]
---
commit 3a26bafa2e27892c9e157720525a094b55748f09
Author: Philipp Reisner philipp.reis...@linbit.com
Date: Tue Aug 25 14:49:19 2009 +0200
Do not consider peer dead when fence-peer handler can not reach it
and we are UpToDate
--- Comment #5 From Philipp Reisner 2009-10-21 13:26:58 [reply]
---
Released with 8.3.3
We setup our metadata in advance before we use drbdsetup to attach
the
disk. Here's the
metadata.
version v08
Re: [DRBD-user] drbd fencing policy problem, upgrading from 8.2.7 - 8.3.6
On Fri, Feb 05, 2010 at 12:46:24PM -0500, Petrakis, Peter wrote:
Hi All,
We use resource-only fencing currently which is causing us a problem
when we're bringing up a cluster for the first time. This all worked
well with 8.2.7. We only have one node active at this time and the
fencing handler is bailing with '5' which is correct. The problem is,
this return failure is stopping us from becoming Primary.
block drbd5: helper command: /usr/lib/spine/bin/avance_drbd_helper
fence-peer minor-5 exit code 5 (0x500)
block drbd5: State change failed: Refusing to be Primary without at
least one UpToDate disk
block drbd5: state = { cs:StandAlone ro:Secondary/Unknown
ds:Consistent/DUnknown r--- }
block drbd5: wanted = { cs:StandAlone ro:Primary/Unknown
ds:Consistent/DUnknown r--- }
I'm quoting our internal bugzilla here:
scenario:
A (Primary) --- b (Secondary)
A (Primary) b (down, crashed; does not know that it is out of date)
a (down)
clean shutdown or crashed, does not matter, knows b is Outdated...
a (still down) b (reboot)
after init dead, heartbeat tries to promote
current behaviour:
b (Secondary, Consistent, pdsk DUnknown)
calls dopd, times out, considers other node as outdated,
and goes Primary.
proposal:
only try to outdate peer if local data is UpToDate.
- works for primary crash (secondary data is UpToDate)
- works for secondary crash (naturally)
- works for primary reboot while secondary offline,
because pdsk is marked Outdated in local meta data,
so we can become UpToDate on restart.
- additionally works for previously described scenario
because crashed secondary has pdsk Unknown,
thus comes up as Consistent (not UpToDate)
avoids one more way to create diverging data sets.
if you want to force Consistent to be UpToDate,
you'd then need either fiddle with drbdmeta,
or maybe just the good old --overwrite-data-of-peer does the trick?
does not work for cluster crash,
when only former secondary comes up.
--- Comment #1 From Florian Haas 2009-07-03 09:07:57 [reply] ---
Bumping severity to normal and setting target milestone to 8.3.3.
This is not an enhancement feature, it's a real (however subtle) bug. This
tends to break dual-Primary setups with fencing, such as with GFS:
- Dual Primary configuration, GFS and CMAN running.
- Replication link goes away.
- Both nodes are now degraded clusters.
- GFS/CMAN initiates fencing, one node gets rebooted.
- Node reboots, link is still down. Since we were degraded clusters to begin
with, degr-wfc-timeout now applies, which is finite by default.
- After the timeout expires, the recovered node is now Consistent and attempts
to fence the peer, when it should not.
- Since the network link is still down, fencing fails, but we now assume the
peer is dead, and the node becomes Primary anyway.
- We have split brain, diverging datasets, all our fencing precautions are moot.
--- Comment #3 From Philipp Reisner 2009-08-25 14:42:58 [reply] ---
Proposed solution:
Just to recap, the expected exit codes of the fence-peer-hander:
3 Peer's disk state was already Inconsistent.
4 Peer's disk state was successfully set to Outdated (or was Outdated to begin
with).
5 Connection to the peer node failed, peer could not be reached.
6 Peer refused to be outdated because the affected resource was in the primary
role.
7 Peer node was successfully fenced off the cluster. This should never occur
unless fencing is set to resource-and-stonith for the affected resource.
Now, if we get a 5 (peer not reachable) and we are not UpToDate (that means we
are only Consistent) then refuse to become primary and do not consider the peer
as outdated.
The change in DRBD is minimal, but we need then to implement exit code 5 also
in crm-fence-peer.sh
--- Comment #4 From Philipp Reisner 2009-08-26 19:15:33 [reply] ---
commit 3a26bafa2e27892c9e157720525a094b55748f09
Author: Philipp Reisner philipp.reis...@linbit.com
Date: Tue Aug 25 14:49:19 2009 +0200
Do not consider peer dead when fence-peer handler can not reach it and we
are UpToDate
--- Comment #5 From Philipp Reisner 2009-10-21 13:26:58 [reply] ---
Released with 8.3.3
We setup our metadata in advance before we use drbdsetup to attach the
disk. Here's the
metadata.
version v08;
uuid {
0x0006; 0x; 0x;
0x;
flags 0x0011;
Right. I assume you do it this way to avoid the initial sync.
There is no need for that anymore, we now have the new-uuid --clear-bitmap
for that.
Anyways, it should work for you again if you initialize the flags to also
mark the peer as outdated, i.e. instead of 0x11, make that 0x31.
The flags we set ought to be telling DRBD that our data is UpToDate
enough to become
Primary but it doesn't seem to matter while fencing is enabled. The
result is
the same regardless of whether I specify '-o' while calling
