On Wed, Feb 29, 2012 at 5:21 AM, William Seligman <[email protected]> wrote: > On 2/27/12 8:40 PM, Andrew Beekhof wrote: > >> Oh, what does the fence_pcmk file look like? > > This is a standard part of the pacemaker-1.1.6 package.
I know, I wrote it :-) I'm just curious exactly what it contains, there was a rather serious bug at one point. > According to > > <http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/_configuring_cman_fencing.html> > > it causes any fencing requests from cman to be redirected to pacemaker. > > Since you asked, I've attached a copy of the file. I note that if this script > is > used to fence a system it writes to /var/log/messages using logger, and there > is > no such log message in my logs. So I guess cman is off the hook. > >> On Tue, Feb 28, 2012 at 11:49 AM, William Seligman >> <[email protected]> wrote: >>> I'm trying to set up an active/active HA cluster as explained in Clusters >>> From >>> Scratch (which I just re-read after my last problem). >>> >>> I'll give versions and config files below, but I'll start with what >>> happens. I >>> start with an active/active cman+pacemaker+drbd+gfs2 cluster, with fencing >>> enabled. My fencing mechanism cuts power to a node by turning the load off >>> in >>> its UPS. The two nodes are hypatia-tb and orestes-tb. >>> >>> I want to test fencing and recovery. I start with both nodes running, and >>> resources properly running on both nodes. Then I simulate failure on one >>> node, >>> e.g., orestes-tb. I've done this with "crm node standby", "service pacemaker >>> off", or by pulling the plug. As expected, all the resources move to >>> hypatia-tb, >>> with the drbd resource as Primary. >>> >>> When I try to bring orestes-tb back into the cluster with "crm node online" >>> or >>> "service pacemaker on" (the inverse of how I removed it), orestes-tb is >>> fenced. >>> OK, that makes sense, I guess; there's a potential split-brain situation. >> >> Not really, that should only happen if the two nodes can't see each >> other. Which should not be the case. >> Only when you pull the plug should orestes-tb be fenced. >> >> Or if you're using a fencing device that requires the node to have >> power, then I can imagine that turning it on again might result in >> fencing. >> But not for the other cases. > > I ran a test: I turned off pacemaker (and so DRBD) on orestes-tb. I "touch"ed > a > file on the hypatia-tb DRBD partition, to make it the "newer" one. Then I > turned > off pacemaker on hypatia-tb. Finally I turned on just drbd on hypatia-tb, then > on orestes-tb. > > From /var/log/messages on hypatia-tb: > > Feb 28 11:39:19 hypatia-tb kernel: d-con admin: Starting worker thread (from > drbdsetup [21822]) > Feb 28 11:39:19 hypatia-tb kernel: block drbd0: disk( Diskless -> Attaching ) > Feb 28 11:39:19 hypatia-tb kernel: d-con admin: Method to ensure write > ordering: > barrier > Feb 28 11:39:19 hypatia-tb kernel: block drbd0: max BIO size = 130560 > Feb 28 11:39:19 hypatia-tb kernel: block drbd0: Adjusting my ra_pages to > backing > device's (32 -> 768) > Feb 28 11:39:19 hypatia-tb kernel: block drbd0: drbd_bm_resize called with > capacity == 5611549368 > Feb 28 11:39:19 hypatia-tb kernel: block drbd0: resync bitmap: bits=701443671 > words=10960058 pages=21407 > Feb 28 11:39:19 hypatia-tb kernel: block drbd0: size = 2676 GB (2805774684 KB) > Feb 28 11:39:19 hypatia-tb kernel: block drbd0: bitmap READ of 21407 pages > took > 576 jiffies > Feb 28 11:39:20 hypatia-tb kernel: block drbd0: recounting of set bits took > additional 87 jiffies > Feb 28 11:39:20 hypatia-tb kernel: block drbd0: 55 MB (14114 bits) marked > out-of-sync by on disk bit-map. > Feb 28 11:39:20 hypatia-tb kernel: block drbd0: disk( Attaching -> UpToDate ) > pdsk( DUnknown -> Outdated ) > Feb 28 11:39:20 hypatia-tb kernel: block drbd0: attached to UUIDs > 862A336609FD27CD:BFFB722D5E3E15D7:6E63EC4258C86AF2:6E62EC4258C86AF2 > Feb 28 11:39:20 hypatia-tb kernel: d-con admin: conn( StandAlone -> > Unconnected ) > Feb 28 11:39:20 hypatia-tb kernel: d-con admin: Starting receiver thread (from > drbd_w_admin [21824]) > Feb 28 11:39:20 hypatia-tb kernel: d-con admin: receiver (re)started > Feb 28 11:39:20 hypatia-tb kernel: d-con admin: conn( Unconnected -> > WFConnection ) > > > From /var/log/messages on orestes-tb: > > Feb 28 11:39:51 orestes-tb kernel: d-con admin: Starting worker thread (from > drbdsetup [17827]) > Feb 28 11:39:51 orestes-tb kernel: block drbd0: disk( Diskless -> Attaching ) > Feb 28 11:39:51 orestes-tb kernel: d-con admin: Method to ensure write > ordering: > barrier > Feb 28 11:39:51 orestes-tb kernel: block drbd0: max BIO size = 130560 > Feb 28 11:39:51 orestes-tb kernel: block drbd0: Adjusting my ra_pages to > backing > device's (32 -> 768) > Feb 28 11:39:51 orestes-tb kernel: block drbd0: drbd_bm_resize called with > capacity == 5611549368 > Feb 28 11:39:51 orestes-tb kernel: block drbd0: resync bitmap: bits=701443671 > words=10960058 pages=21407 > Feb 28 11:39:51 orestes-tb kernel: block drbd0: size = 2676 GB (2805774684 KB) > Feb 28 11:39:52 orestes-tb kernel: block drbd0: bitmap READ of 21407 pages > took > 735 jiffies > Feb 28 11:39:52 orestes-tb kernel: block drbd0: recounting of set bits took > additional 93 jiffies > Feb 28 11:39:52 orestes-tb kernel: block drbd0: 0 KB (0 bits) marked > out-of-sync > by on disk bit-map. > Feb 28 11:39:52 orestes-tb kernel: block drbd0: disk( Attaching -> Outdated ) > Feb 28 11:39:52 orestes-tb kernel: block drbd0: attached to UUIDs > BFFB722D5E3E15D6:0000000000000000:6E63EC4258C86AF2:6E62EC4258C86AF2 > Feb 28 11:39:52 orestes-tb kernel: d-con admin: conn( StandAlone -> > Unconnected ) > Feb 28 11:39:52 orestes-tb kernel: d-con admin: Starting receiver thread (from > drbd_w_admin [17829]) > Feb 28 11:39:52 orestes-tb kernel: d-con admin: receiver (re)started > Feb 28 11:39:52 orestes-tb kernel: d-con admin: conn( Unconnected -> > WFConnection ) > Feb 28 11:39:52 orestes-tb kernel: d-con admin: Handshake successful: Agreed > network protocol version 100 > Feb 28 11:39:52 orestes-tb kernel: d-con admin: conn( WFConnection -> > WFReportParams ) > Feb 28 11:39:52 orestes-tb kernel: d-con admin: Starting asender thread (from > drbd_r_admin [17835]) > Feb 28 11:39:52 orestes-tb kernel: block drbd0: drbd_sync_handshake: > Feb 28 11:39:52 orestes-tb kernel: block drbd0: self > BFFB722D5E3E15D6:0000000000000000:6E63EC4258C86AF2:6E62EC4258C86AF2 bits:0 > flags:0 > Feb 28 11:39:52 orestes-tb kernel: block drbd0: peer > 862A336609FD27CC:BFFB722D5E3E15D7:6E63EC4258C86AF2:6E62EC4258C86AF2 bits:14114 > flags:0 > Feb 28 11:39:52 orestes-tb kernel: block drbd0: uuid_compare()=-1 by rule 50 > Feb 28 11:39:52 orestes-tb kernel: block drbd0: peer( Unknown -> Secondary ) > conn( WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) > Feb 28 11:39:52 orestes-tb kernel: block drbd0: receive bitmap stats > [Bytes(packets)]: plain 0(0), RLE 176(1), total 176; compression: 100.0% > Feb 28 11:39:52 orestes-tb kernel: block drbd0: send bitmap stats > [Bytes(packets)]: plain 0(0), RLE 176(1), total 176; compression: 100.0% > Feb 28 11:39:52 orestes-tb kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID > ) > Feb 28 11:40:01 orestes-tb corosync[2193]: [TOTEM ] A processor failed, > forming new configuration. > Feb 28 11:40:03 orestes-tb corosync[2193]: [QUORUM] Members[1]: 2 > Feb 28 11:40:03 orestes-tb corosync[2193]: [TOTEM ] A processor joined or > left > the membership and a new membership was formed. > Feb 28 11:40:03 orestes-tb kernel: dlm: closing connection to node 1 > Feb 28 11:40:03 orestes-tb corosync[2193]: [CPG ] chosen downlist: sender > r(0) ip(129.236.252.14) r(1) ip(192.168.100.6) ; members(old:2 left:1) > Feb 28 11:40:03 orestes-tb corosync[2193]: [MAIN ] Completed service > synchronization, ready to provide service. > Feb 28 11:40:03 orestes-tb fenced[2247]: fencing node > hypatia-tb.nevis.columbia.edu > > > As far as I can tell, hypatia-tb's drbd comes up, says "I'm UpToDate" and > waits > for a connection from orestes-tb. orestes-tb's drbd comes up, says "I'm > UpToDate" and starts the sync process with hypatia-tb. Then cman+corosync > steps > in on orestes-tb and fences hypatia-tb, before the sync can proceed. > > I ran another test. I did the same thing as the previous paragraph, except > that > I made sure both cman and pacemaker were off (I had to reboot to make sure) > and > just started drbd on both nodes. Sure enough, drbd was able to sync without > split-brain or fencing. So this is a cman/corosync issue, not a drbd issue. > > While I was setting up the test for the previous paragraph, there was a > problem > with another resource (ocf:heartbeat:exportfs) that couldn't be properly > monitored on either node. This led to a cycle of fencing where each node would > successively fence the other because the exportfs resource couldn't run on > either node. I had to quickly change my configuration to turn off monitoring > on > the resource. Not being able to run is fine, but not being able to stop would definitely cause fencing. Make sure the RA can always stop ;-) > > So it seems like cman+corosync is the issue. It's as if I"m "over-fencing." > > Any ideas? > >>> I bring orestes-tb back up, with the intent of adding it back into the >>> cluster. >>> I make sure cman, pacemaker, and drbd services were off at system start. On >>> orestes-tb, I type "service drbd start". >>> >>> What I expect to happen is that the drbd resource on orestes-tb is marked >>> "Outdated" or something like that. Then I'd fix it with "drbdadm >>> --discard-my-data connect admin" or whatever is appropriate. >>> >>> What actually happens is that hypatia-tb is fenced. Since this is the node >>> running all the resources, this is bad behavior. It's even more puzzling >>> when I >>> consider that at, the time, there isn't any fencing resource actually >>> running on >>> orestes-tb; my guess is that DRBD on hypatia-tb is fencing itself. >>> >>> Eventually hypatia-tb reboots, and the cluster goes back to normal. But as a >>> fencing/stability/HA test, this is a failure. >>> >>> I've repeated this with a number of variations. In the end, both systems >>> have to >>> be fenced/rebooted before the cluster is working again. >>> >>> Any ideas? >>> >>> Versions: >>> >>> Scientific Linux 6.2 >>> kernel 2.6.32 >>> cman-3.0.12 >>> corosync-1.4.1 >>> pacemaker-1.1.6 >>> drbd-8.4.1 >>> >>> /etc/drbd.d/global-common.conf: >>> >>> global { >>> � � � �usage-count yes; >>> } >>> >>> common { >>> � � � �startup { >>> � � � � � � � �wfc-timeout � � � � � � 60; >>> � � � � � � � �degr-wfc-timeout � � � �60; >>> � � � � � � � �outdated-wfc-timeout � �60; >>> � � � �} >>> } >>> >>> /etc/drbd.d/admin.res: >>> >>> resource admin { >>> >>> � � � �protocol C; >>> >>> � � � �on hypatia-tb.nevis.columbia.edu { >>> � � � � � � � �volume 0 { >>> � � � � � � � � � � � �device � � � � �/dev/drbd0; >>> � � � � � � � � � � � �disk � � � � � �/dev/md2; >>> � � � � � � � � � � � �flexible-meta-disk � � �internal; >>> � � � � � � � �} >>> � � � � � � � �address � � � � 192.168.100.7:7788; >>> � � � �} >>> � � � �on orestes-tb.nevis.columbia.edu { >>> � � � � � � � �volume 0 { >>> � � � � � � � � � � � �device � � � � �/dev/drbd0; >>> � � � � � � � � � � � �disk � � � � � �/dev/md2; >>> � � � � � � � � � � � �flexible-meta-disk � � �internal; >>> � � � � � � � �} >>> � � � � � � � �address � � � � 192.168.100.6:7788; >>> � � � �} >>> >>> � � � �startup { >>> � � � �} >>> >>> � � � �net { >>> � � � � � � � �allow-two-primaries � � yes; >>> � � � � � � � �after-sb-0pri � � �discard-zero-changes; >>> � � � � � � � �after-sb-1pri � � �discard-secondary; >>> � � � � � � � �after-sb-2pri � � �disconnect; >>> � � � � � � � �sndbuf-size 0; >>> � � � �} >>> >>> � � � �disk { >>> � � � � � � � �resync-rate � � 100M; >>> � � � � � � � �c-max-rate � � �100M; >>> � � � � � � � �al-extents � � �3389; >>> � � � � � � � �fencing resource-only; >>> � � � �} >>> >>> An edited output of "crm configure show": >>> >>> node hypatia-tb.nevis.columbia.edu >>> node orestes-tb.nevis.columbia.edu >>> primitive StonithHypatia stonith:fence_nut \ >>> � params pcmk_host_check="static-list" \ >>> � pcmk_host_list="hypatia-tb.nevis.columbia.edu" \ >>> � ups="sofia-ups" username="admin" password="XXX" >>> primitive StonithOrestes stonith:fence_nut \ >>> � params pcmk_host_check="static-list" \ >>> � pcmk_host_list="orestes-tb.nevis.columbia.edu" >>> � ups="dc-test-stand-ups" username="admin" password="XXX" >>> location StonithHypatiaLocation StonithHypatia \ >>> � -inf: hypatia-tb.nevis.columbia.edu >>> location StonithOrestesLocation StonithOrestes \ >>> � -inf: orestes-tb.nevis.columbia.edu >>> >>> /etc/cluster/cluster.conf: >>> >>> <?xml version="1.0"?> >>> <cluster config_version="17" name="Nevis_HA"> >>> �<logging debug="off"/> >>> �<cman expected_votes="1" two_node="1" /> >>> �<clusternodes> >>> � �<clusternode name="hypatia-tb.nevis.columbia.edu" nodeid="1"> >>> � � �<altname name="hypatia-private.nevis.columbia.edu" port="5405" >>> mcast="226.94.1.1"/> >>> � � �<fence> >>> � � � �<method name="pcmk-redirect"> >>> � � � � �<device name="pcmk" port="hypatia-tb.nevis.columbia.edu"/> >>> � � � �</method> >>> � � �</fence> >>> � �</clusternode> >>> � �<clusternode name="orestes-tb.nevis.columbia.edu" nodeid="2"> >>> � � �<altname name="orestes-private.nevis.columbia.edu" port="5405" >>> mcast="226.94.1.1"/> >>> � � �<fence> >>> � � � �<method name="pcmk-redirect"> >>> � � � � �<device name="pcmk" port="orestes-tb.nevis.columbia.edu"/> >>> � � � �</method> >>> � � �</fence> >>> � �</clusternode> >>> �</clusternodes> >>> �<fencedevices> >>> � �<fencedevice name="pcmk" agent="fence_pcmk"/> >>> �</fencedevices> >>> �<fence_daemon post_join_delay="30" /> >>> �<rm disabled="1" /> >>> </cluster> >>> >>> >>> The log messages on orestes-tb, just before hypatia-tb is fenced (there are >>> no >>> messages in the hypatia-tb log for this time): >>> >>> Feb 15 16:52:27 orestes-tb kernel: drbd: initialized. Version: 8.4.1 >>> (api:1/proto:86-100) >>> Feb 15 16:52:27 orestes-tb kernel: drbd: GIT-hash: >>> 91b4c048c1a0e06777b5f65d312b38d47abaea80 build by >>> [email protected], 2012-02-14 17:05:32 >>> Feb 15 16:52:27 orestes-tb kernel: drbd: registered as block device major >>> 147 >>> Feb 15 16:52:27 orestes-tb kernel: d-con admin: Starting worker thread (from >>> drbdsetup [2570]) >>> Feb 15 16:52:27 orestes-tb kernel: block drbd0: disk( Diskless -> Attaching >>> ) >>> Feb 15 16:52:27 orestes-tb kernel: d-con admin: Method to ensure write >>> ordering: >>> barrier >>> Feb 15 16:52:27 orestes-tb kernel: block drbd0: max BIO size = 130560 >>> Feb 15 16:52:27 orestes-tb kernel: block drbd0: Adjusting my ra_pages to >>> backing >>> device's (32 -> 768) >>> Feb 15 16:52:27 orestes-tb kernel: block drbd0: drbd_bm_resize called with >>> capacity == 5611549368 >>> Feb 15 16:52:27 orestes-tb kernel: block drbd0: resync bitmap: >>> bits=701443671 >>> words=10960058 pages=21407 >>> Feb 15 16:52:27 orestes-tb kernel: block drbd0: size = 2676 GB (2805774684 >>> KB) >>> Feb 15 16:52:28 orestes-tb kernel: block drbd0: bitmap READ of 21407 pages >>> took >>> 634 jiffies >>> Feb 15 16:52:28 orestes-tb kernel: block drbd0: recounting of set bits took >>> additional 92 jiffies >>> Feb 15 16:52:28 orestes-tb kernel: block drbd0: 0 KB (0 bits) marked >>> out-of-sync >>> by on disk bit-map. >>> Feb 15 16:52:28 orestes-tb kernel: block drbd0: disk( Attaching -> Outdated >>> ) >>> Feb 15 16:52:28 orestes-tb kernel: block drbd0: attached to UUIDs >>> F5355FCF6114F218:0000000000000000:8A5519C7090D6BD6:8A5419C7090D6BD6 >>> Feb 15 16:52:28 orestes-tb kernel: d-con admin: conn( StandAlone -> >>> Unconnected ) >>> Feb 15 16:52:28 orestes-tb kernel: d-con admin: Starting receiver thread >>> (from >>> drbd_w_admin [2572]) >>> Feb 15 16:52:28 orestes-tb kernel: d-con admin: receiver (re)started >>> Feb 15 16:52:28 orestes-tb kernel: d-con admin: conn( Unconnected -> >>> WFConnection ) >>> Feb 15 16:52:29 orestes-tb kernel: d-con admin: Handshake successful: Agreed >>> network protocol version 100 >>> Feb 15 16:52:29 orestes-tb kernel: d-con admin: conn( WFConnection -> >>> WFReportParams ) >>> Feb 15 16:52:29 orestes-tb kernel: d-con admin: Starting asender thread >>> (from >>> drbd_r_admin [2579]) >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: drbd_sync_handshake: >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: self >>> F5355FCF6114F218:0000000000000000:8A5519C7090D6BD6:8A5419C7090D6BD6 bits:0 >>> flags:0 >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: peer >>> 06B93A6C54D6D631:F5355FCF6114F219:8A5519C7090D6BD6:8A5419C7090D6BD6 >>> bits:615 flags:0 >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: uuid_compare()=-1 by rule 50 >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: peer( Unknown -> Primary ) >>> conn( >>> WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: receive bitmap stats >>> [Bytes(packets)]: plain 0(0), RLE 39(1), total 39; compression: 100.0% >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: send bitmap stats >>> [Bytes(packets)]: plain 0(0), RLE 39(1), total 39; compression: 100.0% >>> Feb 15 16:52:29 orestes-tb kernel: block drbd0: conn( WFBitMapT -> >>> WFSyncUUID ) >>> Feb 15 16:52:50 orestes-tb kernel: d-con admin: PingAck did not arrive in >>> time. >>> Feb 15 16:52:50 orestes-tb kernel: d-con admin: peer( Primary -> Unknown ) >>> conn( >>> WFSyncUUID -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) >>> Feb 15 16:52:50 orestes-tb kernel: d-con admin: asender terminated >>> Feb 15 16:52:50 orestes-tb kernel: d-con admin: Terminating asender thread >>> Feb 15 16:52:51 orestes-tb kernel: block drbd0: bitmap WRITE of 3 pages >>> took 247 >>> jiffies >>> Feb 15 16:52:51 orestes-tb kernel: block drbd0: 2460 KB (615 bits) marked >>> out-of-sync by on disk bit-map. >>> Feb 15 16:52:51 orestes-tb kernel: d-con admin: Connection closed >>> Feb 15 16:52:51 orestes-tb kernel: d-con admin: conn( NetworkFailure -> >>> Unconnected ) >>> Feb 15 16:52:51 orestes-tb kernel: d-con admin: receiver terminated >>> Feb 15 16:52:51 orestes-tb kernel: d-con admin: Restarting receiver thread >>> Feb 15 16:52:51 orestes-tb kernel: d-con admin: receiver (re)started >>> Feb 15 16:52:51 orestes-tb kernel: d-con admin: conn( Unconnected -> >>> WFConnection ) > > > -- > Bill Seligman | Phone: (914) 591-2823 > Nevis Labs, Columbia Univ | mailto://[email protected] > PO Box 137 | > Irvington NY 10533 USA | http://www.nevis.columbia.edu/~seligman/ > > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
