Hi Andreas, On Sun, Feb 10, 2008 at 09:38:45PM +0100, Andreas Mather1 wrote: > *********************** > Warning: Your file, report_1.tar.gz, contains more than 32 files after > decompression and cannot be scanned. > *********************** > > > > > Hi all, > > Please find attached a hb_report for a problem I experienced when > implementing heartbeat. > > The environment: > It's an asymmetric 4 node cluster, running heartbeat 2.1.3. All nodes share > a couple of filesystems, all GPFS formatted. Services inlcude WebSphere > (modified RA), DB2 (modified RA), vsftpd (Xinetd), samba, nfs, MCS (self > written RA), IHS and are put in 4 groups (filesvc, mcs, was, db). Dejan is > also familiar with the setup. > OS: SLES 9.3 (x86_64) > hearbeat: build via ./ConfigureMe package > > > The Problem: > In general, everything works fine (crm_standby works for every node, etc.), > but, when I simulate a power loss of one node (via IBM RSA)*, a cluster > split occurs when this node rejoins. Suddenly, on every node, crm_mon shows > the node it is running on as 'online' while reporting the other nodes as > 'OFFLINE'. After 1 - 2 min. the cluster is fully operational again (all > nodes found themself again), but it seems as every resource gets restarted. > > Please let me know, if I can provide further information.
>From the log on rbxw02: Feb 10 19:10:16 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth0 up. Feb 10 19:10:16 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth0 up. Feb 10 19:10:32 rbxw02 heartbeat: [22769]: info: Link rbxd02:eth2 dead. Feb 10 19:13:16 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth0 dead. Feb 10 19:13:16 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth2 dead. Feb 10 19:13:17 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth0 dead. Feb 10 19:13:17 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth2 dead. Feb 10 19:15:06 rbxw02 heartbeat: [22769]: CRIT: Cluster node rbxw01 returning after partition. Feb 10 19:15:06 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth2 up. Feb 10 19:15:06 rbxw02 heartbeat: [22769]: info: Link rbxd02:eth2 up. Feb 10 19:15:07 rbxw02 heartbeat: [22769]: CRIT: Cluster node rbxd01 returning after partition. Feb 10 19:15:07 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth2 up. Strange timestamps. Which node went down? And when? Also, rbxd02:eth0 was not reported as down and rbxw01:eth0 rbxd01:eth0 not as up: probably at some point rbxw02:eth0 went down. It would be interesting to see logs from the other nodes. Don't know why hb_report didn't pack them. Two extra nodes went DC around 19:13 for about two minutes, which means that there were three partitions: w02,d02 and w01 and d01. Note that none of them had quorum. Looks like a network problem, but an awkward one. Don't know how it got disrupted this much. Perhaps you could try with unicast: replace each mcast directive with four ucast directives. Cheers, Dejan > Thanks, > > Andreas > > > * Sorry, I forgot to test what happens, when I just stop and start > heartbeat on that node - would be useful too, I think... :( > > > > > (See attached file: report_1.tar.gz) > > Mit freundlichen Gr??en / Best regards > > Andreas MATHER > ESLT - Enterprise Services for Linux Technologies > > IBM Austria, Obere Donaustrasse 95, 1020 Vienna > Phone : +43-1-21145/4799 > Fax: +43-1-21145/8888 > e-mail: [EMAIL PROTECTED] > > IBM ?sterreich Internationale B?romaschinen Gesellschaft m.b.H. > Sitz: Wien > Firmenbuchgericht: Handelsgericht Wien, FN 80000y > _______________________________________________________ > Linux-HA-Dev: [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ _______________________________________________________ Linux-HA-Dev: [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
