On Thu, Mar 01, 2012 at 12:16:17PM -0500, William Seligman wrote: > On 3/1/12 12:10 PM, William Seligman wrote: > > On 3/1/12 6:34 AM, emmanuel segura wrote: > >> try to change the fence daemon tag like this > >> ==================================== > >> <fence_daemon clean_start="1" post_join_delay="30" /> > >> ==================================== > >> change your cluster config version and after reboot the cluster > > > > This did not change the behavior of the cluster. In particular, I'm still > > dealing with this: > > > >>>>> - If the system starts with cman running, and I start drbd, it's > >>>>> likely that system who is _not_ Outdated will be fenced (rebooted). > > This just happened again. Here's the log from the "bad" node, the one I > stopped > and then restarted. cman is running (not pacemaker). I start drbd: > > Mar 1 12:03:49 orestes-tb kernel: drbd: initialized. Version: 8.3.12 > (api:88/proto:86-96) > Mar 1 12:03:49 orestes-tb kernel: drbd: GIT-hash: > e2a8ef4656be026bbae540305fcb998a5991090f build by > [email protected], 2012-02-28 18:01:34 > Mar 1 12:03:49 orestes-tb kernel: drbd: registered as block device major 147 > Mar 1 12:03:49 orestes-tb kernel: drbd: minor_table @ 0xffff88041dbc4b80 > Mar 1 12:03:49 orestes-tb kernel: block drbd0: Starting worker thread (from > cqueue [2942]) > Mar 1 12:03:49 orestes-tb kernel: block drbd0: disk( Diskless -> Attaching ) > Mar 1 12:03:50 orestes-tb kernel: block drbd0: Found 57 transactions (57 > active > extents) in activity log. > Mar 1 12:03:50 orestes-tb kernel: block drbd0: Method to ensure write > ordering: > barrier > Mar 1 12:03:50 orestes-tb kernel: block drbd0: max BIO size = 130560 > Mar 1 12:03:50 orestes-tb kernel: block drbd0: Adjusting my ra_pages to > backing > device's (32 -> 768) > Mar 1 12:03:50 orestes-tb kernel: block drbd0: drbd_bm_resize called with > capacity == 5611549368 > Mar 1 12:03:50 orestes-tb kernel: block drbd0: resync bitmap: bits=701443671 > words=10960058 pages=21407 > Mar 1 12:03:50 orestes-tb kernel: block drbd0: size = 2676 GB (2805774684 KB) > Mar 1 12:03:50 orestes-tb kernel: block drbd0: bitmap READ of 21407 pages > took > 625 jiffies > Mar 1 12:03:50 orestes-tb kernel: block drbd0: recounting of set bits took > additional 86 jiffies > Mar 1 12:03:50 orestes-tb kernel: block drbd0: 0 KB (0 bits) marked > out-of-sync > by on disk bit-map. > Mar 1 12:03:50 orestes-tb kernel: block drbd0: disk( Attaching -> Outdated ) > Mar 1 12:03:50 orestes-tb kernel: block drbd0: attached to UUIDs > 878999EFCFBE8E08:0000000000000000:494B48826E41A2C2:494A48826E41A2C3 > Mar 1 12:03:50 orestes-tb kernel: block drbd0: conn( StandAlone -> > Unconnected ) > Mar 1 12:03:50 orestes-tb kernel: block drbd0: Starting receiver thread (from > drbd0_worker [2951]) > Mar 1 12:03:50 orestes-tb kernel: block drbd0: receiver (re)started > Mar 1 12:03:50 orestes-tb kernel: block drbd0: conn( Unconnected -> > WFConnection ) > Mar 1 12:03:51 orestes-tb kernel: block drbd0: Handshake successful: Agreed > network protocol version 96 > Mar 1 12:03:51 orestes-tb kernel: block drbd0: conn( WFConnection -> > WFReportParams ) > Mar 1 12:03:51 orestes-tb kernel: block drbd0: Starting asender thread (from > drbd0_receiver [2965]) > Mar 1 12:03:51 orestes-tb kernel: block drbd0: data-integrity-alg: <not-used> > Mar 1 12:03:51 orestes-tb kernel: block drbd0: drbd_sync_handshake: > Mar 1 12:03:51 orestes-tb kernel: block drbd0: self > 878999EFCFBE8E08:0000000000000000:494B48826E41A2C2:494A48826E41A2C3 bits:0 > flags:0 > Mar 1 12:03:51 orestes-tb kernel: block drbd0: peer > D40A1613FAE8F5E9:878999EFCFBE8E09:878899EFCFBE8E09:494B48826E41A2C3 bits:0 > flags:0 > Mar 1 12:03:51 orestes-tb kernel: block drbd0: uuid_compare()=-1 by rule 50 > Mar 1 12:03:51 orestes-tb kernel: block drbd0: peer( Unknown -> Primary ) > conn( > WFReportParams -> WFBitMapT ) pdsk( DUnknown -> UpToDate ) > Mar 1 12:03:53 orestes-tb kernel: block drbd0: conn( WFBitMapT -> WFSyncUUID > ) > Mar 1 12:04:01 orestes-tb corosync[2296]: [TOTEM ] A processor failed, > forming new configuration.
some random thoughts... DRBD Bitmap exchange causes congestion on Network, packet storm, irq storm, whatever, and UDP cluster comm packets "falling on the floor"? Can you change your cluster comm to use an (additional?) dedicated link? Or play with (increase) totem timeouts? Or play with some sysctls to make it less likely for UDP to "fall on the floor"; if that is what is happening. Maybe if you tcpdump the traffic while you start things up, that could give you some hints as to why corosync thinks that "A processor failed", and it has to fence that failed processor... > Mar 1 12:04:03 orestes-tb corosync[2296]: [QUORUM] Members[1]: 2 > Mar 1 12:04:03 orestes-tb corosync[2296]: [TOTEM ] A processor > joined or left the membership and a new membership was formed. > Mar 1 12:04:03 orestes-tb kernel: dlm: closing connection to node 1 > Mar 1 12:04:03 orestes-tb corosync[2296]: [CPG ] chosen downlist: sender > r(0) ip(129.236.252.14) r(1) ip(192.168.100.6) ; members(old:2 left:1) > Mar 1 12:04:03 orestes-tb corosync[2296]: [MAIN ] Completed service > synchronization, ready to provide service. > Mar 1 12:04:03 orestes-tb fenced[2350]: fencing node > hypatia-tb.nevis.columbia.edu > > > As near as I can tell, the "bad" node sees that the "good" node is Primary and > UpToDate, goes into WFSyncUUID... and then corosync/cman cheerfully fences the > "good" node. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
