Hi, everibody! I have some problems with my Heartbeat: I have configurated a cluster with heartbeat-1.2.5_6, The array works fine but sometimes the secondary node presents some problems that I have not identified. For example: at specific time, the secondary node detects to the primary node like dead, initiates the execution of STONITH but it fails when trying to reset the primary node. After this, the local service restart and automatically the process of mysql server finishes. I think this problem is for network communication, in particular in the cron configuration I found the execution of peridic daily maintenance at the same time and within this is a process is executed that reviews the network interfaces. Periodic daily was commented and it is not executed, nevertheless today happen the same: the secondary node detect to primary node like dead, it executes STONITH without success and restart local heartbeat and kiil mysql sever. That it can be happening? What is wrong?
Thanks in advance! Here is my ha.cf configuration: #watchdog /usr/sbin/watchdog bcast le1 keepalive 3 warntime 45 deadtime 120 initdead 300 auto_failback off udpport 694 debug 0 node mynodend1.mysite.net node mynodend2.mysite.net respawn hacluster /usr/local/lib/heartbeat/ipfail stonith external /usr/local/etc/ha.d/st_shutdown_rest.cfg ping 192.168.152.3 192.168.152.1 Note: Each node has two network interfaces, le0 with internet an le1 to heartbeat communication. Adds: My OS is: 7.0-RELEASE FreeBSD And here are my logs: ha-log heartbeat: 2010/01/22_03:16:57 WARN: node mynodend1.mysite.net: is dead heartbeat: 2010/01/22_03:16:57 info: Link mynodend1.mysite.net:le1 dead. heartbeat: 2010/01/22_03:16:57 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:16:57 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:16:57 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:16:57 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:16:57 WARN: Exiting STONITH mynodend1.mysite.net process 15850 returned rc 1. heartbeat: 2010/01/22_03:16:57 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:02 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:02 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:02 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:02 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:02 WARN: Exiting STONITH mynodend1.mysite.net process 15852 returned rc 1. heartbeat: 2010/01/22_03:17:02 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:07 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:07 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:07 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:07 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:07 WARN: Exiting STONITH mynodend1.mysite.net process 15904 returned rc 1. heartbeat: 2010/01/22_03:17:07 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:12 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:12 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:12 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:12 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:12 WARN: Exiting STONITH mynodend1.mysite.net process 15929 returned rc 1. heartbeat: 2010/01/22_03:17:12 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:17 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:17 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:17 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:17 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:17 WARN: Exiting STONITH mynodend1.mysite.net process 15956 returned rc 1. heartbeat: 2010/01/22_03:17:17 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:20 info: Link mynodend1.mysite.net:le1 up. heartbeat: 2010/01/22_03:17:22 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:22 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:22 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:22 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:22 WARN: Exiting STONITH mynodend1.mysite.net process 15958 returned rc 1. heartbeat: 2010/01/22_03:17:22 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:27 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:27 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:28 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:28 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:28 WARN: Exiting STONITH mynodend1.mysite.net process 15966 returned rc 1. heartbeat: 2010/01/22_03:17:28 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:33 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:33 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:33 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:33 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:33 WARN: Exiting STONITH mynodend1.mysite.net process 15993 returned rc 1. heartbeat: 2010/01/22_03:17:33 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:37 WARN: Cluster node mynodend1.mysite.net returning after partition. heartbeat: 2010/01/22_03:17:37 WARN: Deadtime value may be too small. heartbeat: 2010/01/22_03:17:37 info: See documentation for information on tuning deadtime. heartbeat: 2010/01/22_03:17:37 WARN: Late heartbeat: Node mynodend1.mysite.net: interval 161289 ms heartbeat: 2010/01/22_03:17:37 info: Status update for node mynodend1.mysite.net: status active heartbeat: 2010/01/22_03:17:37 info: Running /usr/local/etc/ha.d/rc.d/status status heartbeat: 2010/01/22_03:17:38 info: Resetting node mynodend1.mysite.net with [external STONITH device] heartbeat: 2010/01/22_03:17:38 info: Host mynodend1.mysite.net external-reset initiating heartbeat: 2010/01/22_03:17:38 ERROR: command '[email protected] shutdown -r now' failed heartbeat: 2010/01/22_03:17:38 ERROR: Host mynodend1.mysite.net not reset! heartbeat: 2010/01/22_03:17:38 WARN: Exiting STONITH mynodend1.mysite.net process 16063 returned rc 1. heartbeat: 2010/01/22_03:17:38 ERROR: STONITH of mynodend1.mysite.net failed. Retrying... heartbeat: 2010/01/22_03:17:39 info: Heartbeat shutdown in progress. (14293) heartbeat: 2010/01/22_03:17:39 info: Giving up all HA resources. heartbeat: 2010/01/22_03:17:39 info: Releasing resource group: mynodend1.mysite.net 192.168.152.171 mysql-server heartbeat: 2010/01/22_03:17:39 info: Running /usr/local/etc/ha.d/resource.d/mysql-server stop heartbeat: 2010/01/22_03:17:41 info: Running /usr/local/etc/ha.d/resource.d/IPaddr 192.168.152.171 stop heartbeat: 2010/01/22_03:17:41 info: killing /usr/local/lib/heartbeat/ipfail process group 14543 with signal 15 heartbeat: 2010/01/22_03:17:41 info: All HA resources relinquished. heartbeat: 2010/01/22_03:17:41 info: killing hb_giveup_resources process group 16065 with signal 15 heartbeat: 2010/01/22_03:17:42 info: killing HBREAD process 14300 with signal 15 heartbeat: 2010/01/22_03:17:42 info: killing HBWRITE process 14301 with signal 15 heartbeat: 2010/01/22_03:17:42 info: killing HBREAD process 14302 with signal 15 heartbeat: 2010/01/22_03:17:42 info: killing HBFIFO process 14296 with signal 15 heartbeat: 2010/01/22_03:17:42 info: killing HBWRITE process 14297 with signal 15 heartbeat: 2010/01/22_03:17:42 info: killing HBREAD process 14298 with signal 15 heartbeat: 2010/01/22_03:17:42 info: killing HBWRITE process 14299 with signal 15 heartbeat: 2010/01/22_03:17:42 info: Core process 14302 exited. 7 remaining heartbeat: 2010/01/22_03:17:42 info: Core process 14301 exited. 6 remaining heartbeat: 2010/01/22_03:17:42 info: Core process 14298 exited. 5 remaining heartbeat: 2010/01/22_03:17:42 info: Core process 14297 exited. 4 remaining heartbeat: 2010/01/22_03:17:42 info: Core process 14299 exited. 3 remaining heartbeat: 2010/01/22_03:17:42 info: Core process 14296 exited. 2 remaining heartbeat: 2010/01/22_03:17:42 info: Core process 14300 exited. 1 remaining heartbeat: 2010/01/22_03:17:42 info: Heartbeat shutdown complete. heartbeat: 2010/01/22_03:17:42 info: Heartbeat restart triggered. heartbeat: 2010/01/22_03:17:42 info: Restarting heartbeat. heartbeat: 2010/01/22_03:17:42 info: Performing heartbeat restart exec. heartbeat: 2010/01/22_03:19:44 info: Neither logfile nor logfacility found. heartbeat: 2010/01/22_03:19:44 info: Logging defaulting to /var/log/ha-log heartbeat: 2010/01/22_03:19:44 info: ************************** heartbeat: 2010/01/22_03:19:44 info: Configuration validated. Starting heartbeat 1.2.5 heartbeat: 2010/01/22_03:19:44 info: heartbeat: version 1.2.5 heartbeat: 2010/01/22_03:19:44 info: Heartbeat generation: 110 heartbeat: 2010/01/22_03:19:44 info: UDP Broadcast heartbeat started on port 694 (694) interface le1 heartbeat: 2010/01/22_03:19:44 info: ping heartbeat started. heartbeat: 2010/01/22_03:19:44 info: ping heartbeat started. heartbeat: 2010/01/22_03:19:44 info: pid 16711 locked in memory. heartbeat: 2010/01/22_03:19:44 info: Local status now set to: 'up' heartbeat: 2010/01/22_03:19:44 info: pid 16714 locked in memory. heartbeat: 2010/01/22_03:19:44 info: pid 16715 locked in memory. heartbeat: 2010/01/22_03:19:44 info: pid 16716 locked in memory. heartbeat: 2010/01/22_03:19:44 info: Link mynodend2.mysite.net:le1 up. heartbeat: 2010/01/22_03:19:44 info: pid 16717 locked in memory. heartbeat: 2010/01/22_03:19:44 info: pid 16718 locked in memory. heartbeat: 2010/01/22_03:19:44 info: Link 192.168.152.3:192.168.152.3 up. heartbeat: 2010/01/22_03:19:44 info: Status update for node 192.168.152.3: status ping heartbeat: 2010/01/22_03:19:44 info: pid 16719 locked in memory. heartbeat: 2010/01/22_03:19:44 info: Status update for node 192.168.152.1: status ping heartbeat: 2010/01/22_03:19:44 info: pid 16720 locked in memory. heartbeat: 2010/01/22_03:19:44 info: Link 192.168.152.1:192.168.152.1 up. heartbeat: 2010/01/22_03:21:42 info: Link mynodend1.mysite.net:le1 up. heartbeat: 2010/01/22_03:21:42 info: Status update for node mynodend1.mysite.net: status active heartbeat: 2010/01/22_03:21:42 info: Local status now set to: 'active' heartbeat: 2010/01/22_03:21:42 info: Starting child client "/usr/local/lib/heartbeat/ipfail" (275,275) heartbeat: 2010/01/22_03:21:42 info: Checking status of STONITH device [external STONITH device] heartbeat: 2010/01/22_03:21:42 info: Exiting STONITH-stat process 17137 returned rc 0. heartbeat: 2010/01/22_03:21:42 info: Starting "/usr/local/lib/heartbeat/ipfail" as uid 275 gid 275 (pid 17138) heartbeat: 2010/01/22_03:21:42 info: Running /usr/local/etc/ha.d/rc.d/status status heartbeat: 2010/01/22_03:21:44 info: remote resource transition completed. heartbeat: 2010/01/22_03:21:44 info: remote resource transition completed. heartbeat: 2010/01/22_03:21:44 info: Local Resource acquisition completed. (none) heartbeat: 2010/01/22_03:21:44 info: Initial resource acquisition complete (T_RESOURCES(them)) heartbeat: 2010/01/22_03:22:40 WARN: Late heartbeat: Node mynodend1.mysite.net: interval 58859 ms ha-debug heartbeat: 2010/01/22_03:17:37 debug: notify_world: setting SIGCHLD Handler to SIG_DFL heartbeat: 2010/01/22_03:17:39 debug: Starting /usr/local/etc/ha.d/resource.d/mysql-server stop heartbeat: 2010/01/22_03:17:41 debug: /usr/local/etc/ha.d/resource.d/mysql-server stop done. RC=0 heartbeat: 2010/01/22_03:17:41 debug: Starting /usr/local/etc/ha.d/resource.d/IPaddr 192.168.152.171 stop heartbeat: 2010/01/22_03:17:41 debug: /usr/local/etc/ha.d/resource.d/IPaddr 192.168.152.171 stop done. RC=0 heartbeat: 2010/01/22_03:21:42 debug: notify_world: setting SIGCHLD Handler to SIG_DFL mysql-error-log 100122 3:17:39 [Note] /usr/local/libexec/mysqld: Normal shutdown 100122 3:17:39 [Note] Slave I/O thread killed while reading event 100122 3:17:39 [Note] Slave I/O thread exiting, read up to log 'RAND1.002197', position 2495 100122 3:17:39 [Note] Error reading relay log event: slave SQL thread was killed 100122 3:17:39 [Note] /usr/local/libexec/mysqld: Shutdown complete 100122 03:17:39 mysqld ended -- View this message in context: http://old.nabble.com/HA-Linux-problems.-tp27277687p27277687.html Sent from the Linux-HA mailing list archive at Nabble.com. _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
