On 3/1/12 12:56 PM, Lars Ellenberg wrote: > On Thu, Mar 01, 2012 at 12:16:17PM -0500, William Seligman wrote: >> On 3/1/12 12:10 PM, William Seligman wrote: >>> On 3/1/12 6:34 AM, emmanuel segura wrote: >>>> try to change the fence daemon tag like this >>>> ==================================== >>>> <fence_daemon clean_start="1" post_join_delay="30" /> >>>> ==================================== >>>> change your cluster config version and after reboot the cluster >>> >>> This did not change the behavior of the cluster. In particular, I'm still >>> dealing with this: >>> >>>>>>> - If the system starts with cman running, and I start drbd, it's >>>>>>> likely that system who is _not_ Outdated will be fenced (rebooted). >> >> This just happened again. Here's the log from the "bad" node, the one I >> stopped >> and then restarted. cman is running (not pacemaker). I start drbd: >> >> Mar 1 12:03:49 orestes-tb kernel: drbd: initialized. Version: 8.3.12 >> (api:88/proto:86-96) >> Mar 1 12:03:49 orestes-tb kernel: drbd: GIT-hash: >> e2a8ef4656be026bbae540305fcb998a5991090f build by >> [email protected], 2012-02-28 18:01:34 >> Mar 1 12:03:49 orestes-tb kernel: drbd: registered as block device major 147 >> Mar 1 12:03:49 orestes-tb kernel: drbd: minor_table @ 0xffff88041dbc4b80 >> Mar 1 12:03:49 orestes-tb kernel: block drbd0: Starting worker thread (from >> cqueue [2942]) >> Mar 1 12:03:49 orestes-tb kernel: block drbd0: disk( Diskless -> Attaching ) >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: Found 57 transactions (57 >> active >> extents) in activity log. >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: Method to ensure write >> ordering: >> barrier >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: max BIO size = 130560 >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: Adjusting my ra_pages to >> backing >> device's (32 -> 768) >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: drbd_bm_resize called with >> capacity == 5611549368 >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: resync bitmap: bits=701443671 >> words=10960058 pages=21407 >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: size = 2676 GB (2805774684 >> KB) >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: bitmap READ of 21407 pages >> took >> 625 jiffies >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: recounting of set bits took >> additional 86 jiffies >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: 0 KB (0 bits) marked >> out-of-sync >> by on disk bit-map. >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: disk( Attaching -> Outdated ) >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: attached to UUIDs >> 878999EFCFBE8E08:0000000000000000:494B48826E41A2C2:494A48826E41A2C3 >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: conn( StandAlone -> >> Unconnected ) >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: Starting receiver thread >> (from >> drbd0_worker [2951]) >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: receiver (re)started >> Mar 1 12:03:50 orestes-tb kernel: block drbd0: conn( Unconnected -> >> WFConnection ) >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: Handshake successful: Agreed >> network protocol version 96 >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: conn( WFConnection -> >> WFReportParams ) >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: Starting asender thread (from >> drbd0_receiver [2965]) >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: data-integrity-alg: >> <not-used> >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: drbd_sync_handshake: >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: self >> 878999EFCFBE8E08:0000000000000000:494B48826E41A2C2:494A48826E41A2C3 bits:0 >> flags:0 >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: peer >> D40A1613FAE8F5E9:878999EFCFBE8E09:878899EFCFBE8E09:494B48826E41A2C3 bits:0 >> flags:0 >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: uuid_compare()=-1 by rule 50 >> Mar 1 12:03:51 orestes-tb kernel: block drbd0: peer( Unknown -> Primary ) >> conn( >> WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) >> Mar 1 12:03:53 orestes-tb kernel: block drbd0: conn( WFBitMapT -> >> WFSyncUUID ) >> Mar 1 12:04:01 orestes-tb corosync[2296]: [TOTEM ] A processor failed, >> forming new configuration. > > some random thoughts... > > DRBD Bitmap exchange causes congestion on Network, packet storm, irq > storm, whatever, and UDP cluster comm packets "falling on the floor"? > > Can you change your cluster comm to use an (additional?) dedicated link? > Or play with (increase) totem timeouts? Or play with some sysctls to > make it less likely for UDP to "fall on the floor"; if that is what is > happening. > > Maybe if you tcpdump the traffic while you start things up, that could > give you some hints as to why corosync thinks that "A processor failed", > and it has to fence that failed processor... > >> Mar 1 12:04:03 orestes-tb corosync[2296]: [QUORUM] Members[1]: 2 >> Mar 1 12:04:03 orestes-tb corosync[2296]: [TOTEM ] A processor >> joined or left the membership and a new membership was formed. >> Mar 1 12:04:03 orestes-tb kernel: dlm: closing connection to node 1 >> Mar 1 12:04:03 orestes-tb corosync[2296]: [CPG ] chosen downlist: sender >> r(0) ip(129.236.252.14) r(1) ip(192.168.100.6) ; members(old:2 left:1) >> Mar 1 12:04:03 orestes-tb corosync[2296]: [MAIN ] Completed service >> synchronization, ready to provide service. >> Mar 1 12:04:03 orestes-tb fenced[2350]: fencing node >> hypatia-tb.nevis.columbia.edu >> >> >> As near as I can tell, the "bad" node sees that the "good" node is Primary >> and >> UpToDate, goes into WFSyncUUID... and then corosync/cman cheerfully fences >> the >> "good" node.
I've solved the problem. Yes, it was because I did something silly. It was Lars
Ellenberg who provided the key when he suggested I think about communication
between the nodes.
The problem was in cluster.conf. Here's the one that failed:
<?xml version="1.0"?>
<cluster config_version="17" name="Nevis_HA">
<logging debug="off"/>
<cman expected_votes="1" two_node="1" />
<clusternodes>
<clusternode name="hypatia-tb.nevis.columbia.edu" nodeid="1">
<altname name="hypatia-private.nevis.columbia.edu" port="5405"
mcast="226.94.1.1"/>
<fence>
<method name="pcmk-redirect">
<device name="pcmk" port="hypatia-tb.nevis.columbia.edu"/>
</method>
</fence>
</clusternode>
<clusternode name="orestes-tb.nevis.columbia.edu" nodeid="2">
<altname name="orestes-private.nevis.columbia.edu" port="5405"
mcast="226.94.1.1"/>
<fence>
<method name="pcmk-redirect">
<device name="pcmk" port="orestes-tb.nevis.columbia.edu"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="pcmk" agent="fence_pcmk"/>
</fencedevices>
<fence_daemon post_join_delay="30" />
<rm disabled="1" />
</cluster>
The problem is that I was using RRP by putting in the altname tags. I thought I
was being cautious; if cman+corosync couldn't talk over our public network, at
least they could talk over the dedicated 1GB link between the two nodes. This is
the same link DRBD uses.
The cluster.conf that works is:
<?xml version="1.0"?>
<cluster config_version="23" name="Nevis_HA">
<logging debug="off"/>
<cman expected_votes="1" two_node="1" />
<clusternodes>
<clusternode name="hypatia-tb.nevis.columbia.edu" nodeid="1">
<fence>
<method name="pcmk-redirect">
<device name="pcmk" port="hypatia-tb.nevis.columbia.edu"/>
</method>
</fence>
</clusternode>
<clusternode name="orestes-tb.nevis.columbia.edu" nodeid="2">
<fence>
<method name="pcmk-redirect">
<device name="pcmk" port="orestes-tb.nevis.columbia.edu"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="pcmk" agent="fence_pcmk"/>
</fencedevices>
<fence_daemon clean_start="1" post_join_delay="30" />
<rm disabled="1" />
</cluster>
Just take out the altname tags, which turns off RRP.
As a test, since I wasn't sure whether rrp_mode="active" or "passive" was the
default, I tried each of them. Both caused my fencing problem.
I guess the lesson (which I now recall was stated in documentation somewhere) is
not to allow corosync talk over the same link that DRBD is using.
I still wish I could use rrp, but the boxes in my cluster only have two ethernet
ports each. I'll have to see if I can talk my superiors into getting an
expansion card.
--
Bill Seligman | Phone: (914) 591-2823
Nevis Labs, Columbia Univ | mailto://[email protected]
PO Box 137 |
Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
