On 3/15/12 3:45 PM, Vladislav Bogdanov wrote: > 15.03.2012 18:43, William Seligman wrote: >> On 3/15/12 3:43 AM, Vladislav Bogdanov wrote: >>> 14.03.2012 00:42, William Seligman wrote: >>> [snip] >>>> These were the log messages, which show that stonith_admin did its job and >>>> CMAN >>>> was notified of the fencing: <http://pastebin.com/jaH820Bv>. >>> >>> Could you please look at the output of 'dlm_tool ls' and 'dlm_tool dump'? >>> >>> You probably have 'kern_stop' and 'fencing' flags there. That means that >>> dlm is unaware that node is fenced. >> >> Here's 'dlm_tool ls' with both nodes running cman+clvmd+gfs2: >> <http://pastebin.com/QrZtm1Ue> >> >> 'dlm_tool dump': <http://pastebin.com/UKWxx9Y4> >> >> For comparison, I crashed one node and looked at the same output on the >> remaining node: >> dlm_tool ls: <http://pastebin.com/cKVAGxsd> >> dlm_tool dump: <http://pastebin.com/c0h0p22Q> (the post-crash lines begin at >> 1331824940) > > Everything is fine there, dlm correctly understands that node is fenced > and returns to a normal state. > > The only minor issue I see is that fencing took much time - 21 sec.
Hmm. My fencing agent works by toggling the power on a UPS. If all the agent does is "action=off", it will cut power immediately. But if you tell it "action=reboot", it will cut the load, wait 10 seconds, then turn the load back on again; I found I needed that delay because otherwise the UPS might confuse/overlap/ignore sequential commands. Could this be an issue? I've noticed that my fencing agent always seems to be called with "action=reboot" when a node is fenced. Why is it using 'reboot' and not 'off'? Is this the standard, or am I missing a definition somewhere? >> >> I don't see the "kern_stop" or "fencing" flags. There's another thing I don't >> see: at the top of 'dlm_tool dump' it displays most of the contents of my >> cluster.conf file, except for the fencing sections. Here's my cluster.conf >> for >> comparison: <http://pastebin.com/w5XNYyAX> > > It also looks correct (I mean fence_pcmk), but I can be wrong here, I do > not use cman. > >> >> cman doesn't see anything wrong in my cluster.conf file: >> >> # ccs_config_validate >> Configuration validates >> >> But could there be something that's causing the fencing sections to be >> ignored? >> >>>> >>>> Unfortunately, I still got the gfs2 freeze, so this is not the complete >>>> story. >>> >>> Both clvmd and gfs2 use dlm. If dlm layer thinks fencing is not >>> completed, both of them freeze. >> >> I did 'grep -E "(dlm|clvm|fenc)" /var/log/messages' and looked at the time I >> crashed the node: <http://pastebin.com/dvBtdLUs>. I see lines that indicate >> that >> pacemaker and drbd are fencing the node, but nothing from dlm or clvmd. Does >> this indicate what you suggest: Could dlm somehow be ignoring or overlooking >> the >> fencing I put in? Is there any other way to check this? > > No, dlm_controld (and friends) mostly uses different logging method - > that is what you see in dlm_tool dump. -- Bill Seligman | Phone: (914) 591-2823 Nevis Labs, Columbia Univ | mailto://[email protected] PO Box 137 | Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
