Hello! I've been seeing heartbeat cluster problems in Linux-based Vyatta and more recent VyOS networking/router appliances. These are currently based on Debian Squeeze, and thus are using:
Package: heartbeat Version: 1:3.0.3-2 VyOS bug report: http://bugzilla.vyos.net/show_bug.cgi?id=244 The problem is that when there are (unexpected) networking problems causing multicast issues, which cause problems in the inter-cluster communications, the heartbeat processes will die on the cluster nodes, which is bad, right? I assume heartbeat should never die, especially not because of temporary networking issues.. I've also seen heartbeat dying because of temporary network maintenance breaks.. Basicly first I'm seeing this kind of messages: Jun 23 17:55:02 vyos03 heartbeat: [4119]: WARN: node vyos01: is dead Jun 23 17:59:23 vyos03 heartbeat: [4119]: CRIT: Cluster node vyos01 returning after partition. Jun 23 17:59:23 vyos03 heartbeat: [4119]: WARN: Deadtime value may be too small. Jun 23 17:59:23 vyos03 heartbeat: [4119]: WARN: Late heartbeat: Node vyos01: interval 273580 ms Jun 23 17:59:23 vyos03 harc[4961]: info: Running /etc/ha.d//rc.d/status status Jun 23 17:59:25 vyos03 ResourceManager[4991]: info: Releasing resource group: vyos01 IPaddr2-vyatta::10.0.0.10/24/eth1 Jun 23 17:59:25 vyos03 ResourceManager[4991]: info: Running /etc/ha.d/resource.d/IPaddr2-vyatta 10.0.0.10/24/eth1 stop Jun 23 17:59:26 vyos03 heartbeat: [4119]: WARN: 1 lost packet(s) for [vyos01] [421:423] Jun 23 17:59:39 vyos03 heartbeat: [4119]: WARN: Logging daemon is disabled --enabling logging daemon is recommended Jun 23 17:59:40 vyos03 harc[5102]: info: Running /etc/ha.d//rc.d/status status Which seem normal in the case of networking problem.. But then later: Jun 23 19:31:22 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (494 messages in queue) Jun 23 19:31:22 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (495 messages in queue) Jun 23 19:31:23 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (496 messages in queue) Jun 23 19:31:24 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (497 messages in queue) Jun 23 19:31:24 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (498 messages in queue) Jun 23 19:31:25 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (499 messages in queue) Jun 23 19:31:26 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (500 messages in queue) Jun 23 19:31:42 vyos03 heartbeat: last message repeated 25 times The "hist queue" size keeps increasing, and when it gets to 500 messages bad things start happening.. Jun 23 19:31:43 vyos03 heartbeat: [10921]: ERROR: Message hist queue is filling up (500 messages in queue) Jun 23 19:31:49 vyos03 heartbeat: last message repeated 9 times Jun 23 19:31:49 vyos03 heartbeat: [10921]: ERROR: lowseq cannnot be greater than ackseq Jun 23 19:31:50 vyos03 heartbeat: [10923]: CRIT: Emergency Shutdown: Master Control process died. Jun 23 19:31:50 vyos03 heartbeat: [10923]: CRIT: Killing pid 10921 with SIGTERM Jun 23 19:31:50 vyos03 heartbeat: [10923]: CRIT: Killing pid 10924 with SIGTERM Jun 23 19:31:50 vyos03 heartbeat: [10923]: CRIT: Killing pid 10925 with SIGTERM Jun 23 19:31:50 vyos03 heartbeat: [10923]: CRIT: Emergency Shutdown(MCP dead): Killing ourselves. At this point clustering has failed, because the heartbeat services/processes aren't running anymore.. Has anyone else seen this? It seems the bug gets triggered at 500 messages in the hist queue, and then I always see the "ERROR: lowseq cannnot be greater than ackseq" and then heartbeat dies.. Thanks, -- Pasi _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
