OR
(2) If the LAN recovers before a DC election doesn't finish on the stand-by
node,
one node keeps its status as OFFLINE despite the LAN is normal condition.
You said, (2) is a CCM bug, and (1) seems to be the same, so the CCM
"instance" goes backward.
Please see the attached debug-log of case (1) including "debug 1".
I'm also going to post it to a bugzilla.
Thanks,
Junko Ikeda
*** excerpt from ha-log ***
# cat guest1-ha-log | grep -e cib.*ccm -e cib.*mem_handle_event | grep -v
engine
cib[27157]: 2007/04/18_17:59:56 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[27157]: 2007/04/18_17:59:56 info: mem_handle_event: instance=2, nodes=2,
new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
cib[27157]: 2007/04/18_17:59:56 info: cib_ccm_msg_callback: PEER: guest2
cib[27157]: 2007/04/18_17:59:56 info: cib_ccm_msg_callback: PEER: guest1
cib[27157]: 2007/04/18_18:02:17 info: mem_handle_event: Got an event
OC_EV_MS_NOT_PRIMARY from ccm
cib[27157]: 2007/04/18_18:02:17 info: mem_handle_event: instance=2, nodes=2,
new=2, lost=0, n_idx=0, new_idx=0, old_idx=4
cib[27157]: 2007/04/18_18:02:27 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[27157]: 2007/04/18_18:02:27 info: mem_handle_event: no mbr_track info
cib[27157]: 2007/04/18_18:02:27 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[27157]: 2007/04/18_18:02:27 info: mem_handle_event: instance=3, nodes=1,
new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
cib[27157]: 2007/04/18_18:02:27 info: cib_ccm_msg_callback: LOST: guest2
cib[27157]: 2007/04/18_18:02:27 info: cib_ccm_msg_callback: PEER: guest1
cib[27157]: 2007/04/18_18:02:38 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[27157]: 2007/04/18_18:02:38 info: mem_handle_event: no mbr_track info
cib[27157]: 2007/04/18_18:02:38 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[27157]: 2007/04/18_18:02:38 info: mem_handle_event: instance=2, nodes=2,
new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
cib[27157]: 2007/04/18_18:02:38 info: cib_ccm_msg_callback: PEER: guest2
cib[27157]: 2007/04/18_18:02:38 info: cib_ccm_msg_callback: PEER: guest1
cib[27157]: 2007/04/18_18:07:16 info: mem_handle_event: Got an event
OC_EV_MS_NOT_PRIMARY from ccm
cib[27157]: 2007/04/18_18:07:16 info: mem_handle_event: instance=2, nodes=2,
new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
cib[27157]: 2007/04/18_18:07:26 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[27157]: 2007/04/18_18:07:26 info: mem_handle_event: no mbr_track info
cib[27157]: 2007/04/18_18:07:26 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[27157]: 2007/04/18_18:07:26 info: mem_handle_event: instance=3, nodes=1,
new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
cib[27157]: 2007/04/18_18:07:26 info: cib_ccm_msg_callback: LOST: guest2
cib[27157]: 2007/04/18_18:07:26 info: cib_ccm_msg_callback: PEER: guest1
cib[27157]: 2007/04/18_18:07:47 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[27157]: 2007/04/18_18:07:47 info: mem_handle_event: no mbr_track info
cib[27157]: 2007/04/18_18:07:47 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[27157]: 2007/04/18_18:07:47 info: mem_handle_event: instance=2, nodes=2,
new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
cib[27157]: 2007/04/18_18:07:47 info: cib_ccm_msg_callback: PEER: guest2
cib[27157]: 2007/04/18_18:07:47 info: cib_ccm_msg_callback: PEER: guest1
# cat guest2-ha-log | grep -e cib.*ccm -e cib.*mem_handle_event | grep -v
engine
cib[17757]: 2007/04/18_18:01:05 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:01:05 info: mem_handle_event: instance=1, nodes=1,
new=1, lost=0, n_idx=0, new_idx=0, old_idx=3
cib[17757]: 2007/04/18_18:01:05 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:01:07 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17757]: 2007/04/18_18:01:07 info: mem_handle_event: no mbr_track info
cib[17757]: 2007/04/18_18:01:07 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:01:07 info: mem_handle_event: instance=2, nodes=2,
new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
cib[17757]: 2007/04/18_18:01:07 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:01:07 info: cib_ccm_msg_callback: PEER: guest1
cib[17757]: 2007/04/18_18:03:29 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17757]: 2007/04/18_18:03:29 info: mem_handle_event: no mbr_track info
cib[17757]: 2007/04/18_18:03:29 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:03:29 info: mem_handle_event: instance=3, nodes=1,
new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
cib[17757]: 2007/04/18_18:03:29 info: cib_ccm_msg_callback: LOST: guest1
cib[17757]: 2007/04/18_18:03:29 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:03:48 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17757]: 2007/04/18_18:03:48 info: mem_handle_event: no mbr_track info
cib[17757]: 2007/04/18_18:03:48 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:03:48 info: mem_handle_event: instance=1, nodes=1,
new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
cib[17757]: 2007/04/18_18:03:48 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:03:49 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17757]: 2007/04/18_18:03:49 info: mem_handle_event: no mbr_track info
cib[17757]: 2007/04/18_18:03:49 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:03:49 info: mem_handle_event: instance=2, nodes=2,
new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
cib[17757]: 2007/04/18_18:03:49 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:03:49 info: cib_ccm_msg_callback: PEER: guest1
cib[17757]: 2007/04/18_18:08:27 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17757]: 2007/04/18_18:08:27 info: mem_handle_event: no mbr_track info
cib[17757]: 2007/04/18_18:08:27 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:08:27 info: mem_handle_event: instance=3, nodes=1,
new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
cib[17757]: 2007/04/18_18:08:27 info: cib_ccm_msg_callback: LOST: guest1
cib[17757]: 2007/04/18_18:08:27 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:08:57 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17757]: 2007/04/18_18:08:57 info: mem_handle_event: no mbr_track info
cib[17757]: 2007/04/18_18:08:57 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:08:57 info: mem_handle_event: instance=1, nodes=1,
new=0, lost=0, n_idx=0, new_idx=1, old_idx=3
cib[17757]: 2007/04/18_18:08:57 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:08:58 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17757]: 2007/04/18_18:08:58 info: mem_handle_event: no mbr_track info
cib[17757]: 2007/04/18_18:08:58 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17757]: 2007/04/18_18:08:58 info: mem_handle_event: instance=2, nodes=2,
new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
cib[17757]: 2007/04/18_18:08:58 info: cib_ccm_msg_callback: PEER: guest2
cib[17757]: 2007/04/18_18:08:58 info: cib_ccm_msg_callback: PEER: guest1
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Andrew Beekhof
Sent: Tuesday, April 17, 2007 4:35 PM
To: High-Availability Linux Development List
Subject: Re: [Linux-ha-dev] "start-delay" parameter for monitor
operationand"Split-Brain"
On 4/13/07, 池田淳子 <[EMAIL PROTECTED]> wrote:
> Hi Andres, Alan
>
> Thank you for your comments.
> I try to run Dummy resource using these parameters this time.
> <timeout="60" start_delay="10">
> Heartbeat version is 2.0.8.
>
> case01;
> Heartbeat run well on 2 nodes, and I brought down the interconnect LAN
from
> network switch. (shutdown the port)
> Split-Brain has come. Stan-by node would be a DC. I brought up the LAN
> immediately.
> If " Action Dummy01_monitor_10000 " wasn't confirmed on the former
stand-by
> node when the LAN was up, that node would do some strange behavior at the
> next Split-Brain.
I'm lost... what are you talking about here?
I also don't see anything like this in the logs
>
> I can see the following message after I brought down the LAN again...
>
> WARN: do_dc_join_finalize: join-2: We are still in a transition. Delaying
> until the TE completes.
>
> It seems that one node tries to join something, but it keeps failing.
> Is this the correct behavior of Heartbeat2?
it looks like a bug
any chance you could add "debug 1" to ha.cf and reproduce it?
ideally with the latest development version which should be released soonish
>
> case02;
> I did down/up the interconnect LAN again.
> In this case, LAN trouble would be recovered before the DC election on
> stand-by node. After recovering, the former stand-by nodes keeps its
status
> as OFFLINE.
well they didnt recover then did they :-)
this is without doubt a CCM bug
if one greps for the following patterns:
# grep -e cib.*ccm -e cib.*mem_handle_event
split-brain2/case01/dl380g5a-ha-log
then you can clearly see the CCM "instance" going backwards from 3 to 2.
cib[17889]: 2007/04/13_11:24:49 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17889]: 2007/04/13_11:24:49 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17889]: 2007/04/13_11:24:49 info: mem_handle_event: instance=3,
nodes=1, new=0, lost=1, n_idx=0, new_idx=1, old_idx=3
cib[17889]: 2007/04/13_11:24:49 info: cib_ccm_msg_callback: LOST: dl380g5b
cib[17889]: 2007/04/13_11:24:49 info: cib_ccm_msg_callback: PEER: dl380g5a
cib[17889]: 2007/04/13_11:26:08 info: mem_handle_event: Got an event
OC_EV_MS_INVALID from ccm
cib[17889]: 2007/04/13_11:26:08 info: mem_handle_event: Got an event
OC_EV_MS_NEW_MEMBERSHIP from ccm
cib[17889]: 2007/04/13_11:26:08 info: mem_handle_event: instance=2,
nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4
cib[17889]: 2007/04/13_11:26:08 info: cib_ccm_msg_callback: PEER: dl380g5b
cib[17889]: 2007/04/13_11:26:08 info: cib_ccm_msg_callback: PEER: dl380g5a
likewise in split-brain2/case02/dl380g5b-ha-log you can see the
progression 1->2->3->1->2
> Alan, see the attached logs, for details.
>
> Best Regards,
> Junko Ikeda
>
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of 池田淳子
> Sent: Thursday, April 12, 2007 8:16 PM
> To: [email protected]
> Subject: [Linux-ha-dev] "start-delay" parameter for monitor operation
> and"Split-Brain"
>
> Hi all,
>
> I'm newbie, and trying to understand how or when "Split-Brain" happens.
> As trial, I run "Dummy" resource for now.
> Heartbeat version is 2.0.8,
> cib.xml, ha.cf and ha-log are attached.
> See below cases, please give some advice.
>
> *** case 01 ***
> My cib.xml was created using hb_gui, so start-delay was "1m".
> I run Heartbeat on 2 nodes at first, and disconnected the interconnect LAN
> on DC node to lead "Split-Brain". (# ifdown eth2)
> After making "Split-Brain", I would up the LAN with "ifup eth2".
> If eth2 is restored after "Action Dummy01_monitor_10000 (x) confirmed" on
> the former stand-by node, it seems that everything works well.
> But if it's done before the confirmation, there is something wrong.
> In wrong case, I couldn't generate "Split-Brain" again.
> I found that pengine/tengine run on both nodes, but one node kept trying
to
> be DC and Dummy resource wouldn't start on that node, additionally,
> failcount was incremented on that strange node.
>
> log message is here...
> WARN: do_dc_join_finalize: join-3: We are still in a transition. Delaying
> until the TE completes.
>
> in this case, I couldn't shutdown Heartbeat process without KILL
command...
>
> *** case 02 ***
> I changed start-delay from 1m to 0s.
> The confirmation process for monitor would work immediately, so though I
put
> ifdown/ifup in a row, it didn't matter.
>
> Q1; My guess, if the interconnect LAN is down/up in a raw and some
> operations aren't confirmed, one node would consider this situation as
> ERROR, so update its failcaount. This case might appear when "start-delay"
> is long (ex, "1m"), and cause some strange "Split-Brain".
> Is this relationship between "start-delay" and "Split-Brain" correct?
>
> *** case 03 ***
> In case 02, the interconnect LAN is up after "Split-Brain" is completed,
it
> means each node is voted as DC.
> For third test, I tried to down/up the interconnect LAN before the DC
> election didn't finish.
> In the result, I met "Split-Brain" again but one node stayed OFFLINE when
> the interconnect LAN was up.
>
> Q2; I know, I had better to set up two or more interconnect LANs just in
> case, but are there any prefer ways to avoid case 03? ex, tuning some
> parameters or something like that.
>
>
> Best Regards,
> Junko Ikeda
>
> NTT DATA INTELLILINK CORPORATION
> Open Source Solutions Business Unit
> Open Source Business Division
>
> Toyosu Center Building Annex, 3-3-9, Toyosu,
> Koto-ku, Tokyo 135-0061, Japan
> TEL : +81-3-3534-4811
> FAX : +81-3-3534-4814
> mailto:[EMAIL PROTECTED]
> http://www.intellilink.co.jp/
>
> _______________________________________________________
> Linux-HA-Dev: [email protected]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
>
>
>
_______________________________________________________
Linux-HA-Dev: [email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
_______________________________________________________
Linux-HA-Dev: [email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
