ResourceManager[15382]:	2011/09/15_16:12:46 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:46 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:46 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:47 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:47 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:47 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:48 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:48 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:48 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:49 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:49 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:49 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:50 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:50 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:51 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 ERROR: Cannot locate resource script mysql
status: missing job name
Try `status --help' for more information.
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 ERROR: Resource script for mysql probably not LSB-compliant.
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 WARN: it (mysql) MUST succeed on a stop when already stopped
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 WARN: Machine reboot narrowly avoided!
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[15382]:	2011/09/15_16:12:52 info: Running /etc/ha.d/resource.d/IPaddr 10.10.10.31/255.255.255.0 stop
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[15788]:	2011/09/15_16:12:52 INFO:  Success
INFO:  Success
Sep 15 16:12:52 staging1.dev.applepeak.com heartbeat: [15369]: info: All HA resources relinquished.
Sep 15 16:12:53 staging1.dev.applepeak.com heartbeat: [14390]: info: killing /usr/lib64/heartbeat/dopd process group 14420 with signal 15
Sep 15 16:12:54 staging1.dev.applepeak.com heartbeat: [14390]: info: killing HBFIFO process 14395 with signal 15
Sep 15 16:12:54 staging1.dev.applepeak.com heartbeat: [14390]: info: killing HBWRITE process 14396 with signal 15
Sep 15 16:12:54 staging1.dev.applepeak.com heartbeat: [14390]: info: killing HBREAD process 14397 with signal 15
Sep 15 16:12:54 staging1.dev.applepeak.com heartbeat: [14390]: info: Core process 14396 exited. 3 remaining
Sep 15 16:12:54 staging1.dev.applepeak.com heartbeat: [14390]: info: Core process 14397 exited. 2 remaining
Sep 15 16:12:54 staging1.dev.applepeak.com heartbeat: [14390]: info: Core process 14395 exited. 1 remaining
Sep 15 16:12:54 staging1.dev.applepeak.com heartbeat: [14390]: info: staging1.dev.applepeak.com Heartbeat shutdown complete.
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15913]: info: Pacemaker support: false
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15913]: WARN: logd is enabled but logfile/debugfile/logfacility is still configured in ha.cf
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15913]: info: **************************
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15913]: info: Configuration validated. Starting heartbeat 3.0.4
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: heartbeat: version 3.0.4
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: Heartbeat generation: 1316090372
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: Local status now set to: 'up'
Sep 15 16:13:34 staging1.dev.applepeak.com heartbeat: [15914]: info: Link staging1.dev.applepeak.com:eth1 up.
Sep 15 16:13:45 staging1.dev.applepeak.com heartbeat: [15914]: info: Link staging2.dev.applepeak.com:eth1 up.
Sep 15 16:13:45 staging1.dev.applepeak.com heartbeat: [15914]: info: Status update for node staging2.dev.applepeak.com: status up
Sep 15 16:13:45 staging1.dev.applepeak.com heartbeat: [15923]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
harc[15923]:	2011/09/15_16:13:45 info: Running /etc/ha.d//rc.d/status status
Sep 15 16:13:46 staging1.dev.applepeak.com heartbeat: [15914]: debug: get_delnodelist: delnodelist= 
Sep 15 16:13:46 staging1.dev.applepeak.com heartbeat: [15914]: info: Comm_now_up(): updating status to active
Sep 15 16:13:46 staging1.dev.applepeak.com heartbeat: [15914]: info: Local status now set to: 'active'
Sep 15 16:13:46 staging1.dev.applepeak.com heartbeat: [15914]: info: Starting child client "/usr/lib64/heartbeat/dopd" (498,498)
Sep 15 16:13:46 staging1.dev.applepeak.com heartbeat: [15942]: info: Starting "/usr/lib64/heartbeat/dopd" as uid 498  gid 498 (pid 15942)
Sep 15 16:13:46 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [15942]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Sep 15 16:13:46 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [15942]: debug: PID=15942
Sep 15 16:13:46 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [15942]: debug: Signing in with heartbeat
Sep 15 16:13:46 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [15942]: debug: [We are staging1.dev.applepeak.com]
Sep 15 16:13:46 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [15942]: debug: Setting message filter mode
Sep 15 16:13:46 staging1.dev.applepeak.com heartbeat: [15914]: info: Status update for node staging2.dev.applepeak.com: status active
Sep 15 16:13:46 staging1.dev.applepeak.com heartbeat: [15945]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 15 16:13:46 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [15942]: debug: Setting message signal
Sep 15 16:13:46 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [15942]: debug: Waiting for messages...
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
harc[15945]:	2011/09/15_16:13:46 info: Running /etc/ha.d//rc.d/status status
Sep 15 16:13:57 staging1.dev.applepeak.com heartbeat: [15914]: info: remote resource transition completed.
Sep 15 16:13:57 staging1.dev.applepeak.com heartbeat: [15914]: info: remote resource transition completed.
Sep 15 16:13:57 staging1.dev.applepeak.com heartbeat: [15914]: info: Initial resource acquisition complete (T_RESOURCES(us))
IPaddr[16000]:	2011/09/15_16:13:57 INFO:  Resource is stopped
Sep 15 16:13:57 staging1.dev.applepeak.com heartbeat: [15964]: info: Local Resource acquisition completed.
Sep 15 16:13:57 staging1.dev.applepeak.com heartbeat: [15914]: debug: StartNextRemoteRscReq(): child count 1
Sep 15 16:13:57 staging1.dev.applepeak.com heartbeat: [16041]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
harc[16041]:	2011/09/15_16:13:57 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ip-request-resp[16041]:	2011/09/15_16:13:57 received ip-request-resp IPaddr::10.10.10.31/255.255.255.0 OK yes
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 info: Acquiring resource group: staging1.dev.applepeak.com IPaddr::10.10.10.31/255.255.255.0 mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[16095]:	2011/09/15_16:13:57 INFO:  Resource is stopped
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 info: Running /etc/ha.d/resource.d/IPaddr 10.10.10.31/255.255.255.0 start
Converted dotted-quad netmask to CIDR as: 24
/usr/lib/ocf/resource.d/heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[16176]:	2011/09/15_16:13:57 INFO: Using calculated nic for 10.10.10.31: eth0
/usr/lib/ocf/resource.d/heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[16176]:	2011/09/15_16:13:57 INFO: eval ifconfig eth0:0 10.10.10.31 netmask 255.255.255.0 broadcast 10.10.10.255
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[16153]:	2011/09/15_16:13:57 INFO:  Success
INFO:  Success
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 ERROR: Cannot locate resource script mysql
status: missing job name
Try `status --help' for more information.
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 CRIT: Giving up resources due to failure of mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 info: Releasing resource group: staging1.dev.applepeak.com IPaddr::10.10.10.31/255.255.255.0 mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:57 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:58 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:58 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:58 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:13:59 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:00 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:00 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:01 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:01 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:01 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:02 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:02 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:02 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:03 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:03 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:03 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:04 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:04 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:04 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:05 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:05 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:05 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:06 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:06 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:06 ERROR: Cannot locate resource script mysql
ARPING 10.10.10.31 from 10.10.10.31 eth0
Sent 10 probes (10 broadcast(s))
Received 0 response(s)
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:07 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:07 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:07 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 info: Retrying failed stop operation [mysql]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 ERROR: Cannot locate resource script mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 ERROR: Cannot locate resource script mysql
status: missing job name
Try `status --help' for more information.
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 ERROR: Resource script for mysql probably not LSB-compliant.
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 WARN: it (mysql) MUST succeed on a stop when already stopped
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 WARN: Machine reboot narrowly avoided!
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[16066]:	2011/09/15_16:14:08 info: Running /etc/ha.d/resource.d/IPaddr 10.10.10.31/255.255.255.0 stop
/usr/lib/ocf/resource.d/heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[16727]:	2011/09/15_16:14:08 INFO: ifconfig eth0:0 down
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[16704]:	2011/09/15_16:14:08 INFO:  Success
INFO:  Success
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
hb_standby[16766]:	2011/09/15_16:14:38 Going standby [foreign].
Sep 15 16:14:38 staging1.dev.applepeak.com heartbeat: [15914]: info: staging1.dev.applepeak.com wants to go standby [foreign]
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [15914]: info: standby: staging2.dev.applepeak.com can take our foreign resources
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [16782]: info: give up foreign HA resources (standby).
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [16782]: info: foreign HA resource release completed (standby).
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [15914]: info: Local standby process completed [foreign].
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [15914]: WARN: 1 lost packet(s) for [staging2.dev.applepeak.com] [41:43]
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [15914]: info: remote resource transition completed.
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [15914]: info: No pkts missing from staging2.dev.applepeak.com!
Sep 15 16:14:39 staging1.dev.applepeak.com heartbeat: [15914]: info: Other node completed standby takeover of foreign resources.
Sep 15 16:18:54 staging1.dev.applepeak.com heartbeat: [17024]: info: Pacemaker support: false
Sep 15 16:18:54 staging1.dev.applepeak.com heartbeat: [17024]: WARN: logd is enabled but logfile/debugfile/logfacility is still configured in ha.cf
Sep 15 16:18:54 staging1.dev.applepeak.com heartbeat: [17024]: info: **************************
Sep 15 16:18:54 staging1.dev.applepeak.com heartbeat: [17024]: info: Configuration validated. Starting heartbeat 3.0.4
Sep 15 16:18:54 staging1.dev.applepeak.com heartbeat: [17024]: info: heartbeat: already running [pid 15914].
Sep 15 16:20:28 staging1.dev.applepeak.com heartbeat: [15914]: info: Heartbeat shutdown in progress. (15914)
Sep 15 16:20:28 staging1.dev.applepeak.com heartbeat: [17103]: info: Giving up all HA resources.
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17116]:	2011/09/15_16:20:28 info: Releasing resource group: staging1.dev.applepeak.com IPaddr::10.10.10.31/255.255.255.0 mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17116]:	2011/09/15_16:20:28 info: Running /etc/ha.d/resource.d/mysql  stop
/etc/ha.d/resource.d/mysql: line 3: This: command not found
/etc/ha.d/resource.d/mysql: line 5: Mar: command not found
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
mysql[17150]:	2011/09/15_16:20:28 Stopping mysqld: [ OK ]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17116]:	2011/09/15_16:20:28 info: Running /etc/ha.d/resource.d/IPaddr 10.10.10.31/255.255.255.0 stop
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[17209]:	2011/09/15_16:20:28 INFO:  Success
INFO:  Success
Sep 15 16:20:28 staging1.dev.applepeak.com heartbeat: [17103]: info: All HA resources relinquished.
Sep 15 16:20:28 staging1.dev.applepeak.com heartbeat: [15914]: WARN: 1 lost packet(s) for [staging2.dev.applepeak.com] [219:221]
Sep 15 16:20:28 staging1.dev.applepeak.com heartbeat: [15914]: info: No pkts missing from staging2.dev.applepeak.com!
Sep 15 16:20:29 staging1.dev.applepeak.com heartbeat: [15914]: info: killing /usr/lib64/heartbeat/dopd process group 15942 with signal 15
Sep 15 16:20:31 staging1.dev.applepeak.com heartbeat: [15914]: info: killing HBFIFO process 15919 with signal 15
Sep 15 16:20:31 staging1.dev.applepeak.com heartbeat: [15914]: info: killing HBWRITE process 15920 with signal 15
Sep 15 16:20:31 staging1.dev.applepeak.com heartbeat: [15914]: info: killing HBREAD process 15921 with signal 15
Sep 15 16:20:31 staging1.dev.applepeak.com heartbeat: [15914]: info: Core process 15919 exited. 3 remaining
Sep 15 16:20:31 staging1.dev.applepeak.com heartbeat: [15914]: info: Core process 15920 exited. 2 remaining
Sep 15 16:20:31 staging1.dev.applepeak.com heartbeat: [15914]: info: Core process 15921 exited. 1 remaining
Sep 15 16:20:31 staging1.dev.applepeak.com heartbeat: [15914]: info: staging1.dev.applepeak.com Heartbeat shutdown complete.
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17334]: info: Pacemaker support: false
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17334]: WARN: logd is enabled but logfile/debugfile/logfacility is still configured in ha.cf
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17334]: info: **************************
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17334]: info: Configuration validated. Starting heartbeat 3.0.4
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: heartbeat: version 3.0.4
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: Heartbeat generation: 1316090373
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth1
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth1 - Status: 1
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: G_main_add_TriggerHandler: Added signal manual handler
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: Local status now set to: 'up'
Sep 15 16:21:11 staging1.dev.applepeak.com heartbeat: [17335]: info: Link staging1.dev.applepeak.com:eth1 up.
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17335]: info: Link staging2.dev.applepeak.com:eth1 up.
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17335]: info: Status update for node staging2.dev.applepeak.com: status up
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17346]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
harc[17346]:	2011/09/15_16:21:34 info: Running /etc/ha.d//rc.d/status status
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17335]: info: Comm_now_up(): updating status to active
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17335]: info: Local status now set to: 'active'
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17335]: info: Starting child client "/usr/lib64/heartbeat/dopd" (498,498)
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17335]: debug: get_delnodelist: delnodelist= 
Sep 15 16:21:34 staging1.dev.applepeak.com heartbeat: [17365]: info: Starting "/usr/lib64/heartbeat/dopd" as uid 498  gid 498 (pid 17365)
Sep 15 16:21:34 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [17365]: WARN: Initializing connection to logging daemon failed. Logging daemon may not be running
Sep 15 16:21:34 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [17365]: debug: PID=17365
Sep 15 16:21:34 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [17365]: debug: Signing in with heartbeat
Sep 15 16:21:34 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [17365]: debug: [We are staging1.dev.applepeak.com]
Sep 15 16:21:34 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [17365]: debug: Setting message filter mode
Sep 15 16:21:35 staging1.dev.applepeak.com heartbeat: [17335]: info: Status update for node staging2.dev.applepeak.com: status active
Sep 15 16:21:35 staging1.dev.applepeak.com heartbeat: [17368]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
Sep 15 16:21:35 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [17365]: debug: Setting message signal
Sep 15 16:21:35 staging1.dev.applepeak.com /usr/lib64/heartbeat/dopd: [17365]: debug: Waiting for messages...
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
harc[17368]:	2011/09/15_16:21:35 info: Running /etc/ha.d//rc.d/status status
Sep 15 16:21:45 staging1.dev.applepeak.com heartbeat: [17335]: info: remote resource transition completed.
Sep 15 16:21:45 staging1.dev.applepeak.com heartbeat: [17335]: info: remote resource transition completed.
Sep 15 16:21:45 staging1.dev.applepeak.com heartbeat: [17335]: info: Initial resource acquisition complete (T_RESOURCES(us))
IPaddr[17422]:	2011/09/15_16:21:45 INFO:  Resource is stopped
Sep 15 16:21:45 staging1.dev.applepeak.com heartbeat: [17386]: info: Local Resource acquisition completed.
Sep 15 16:21:45 staging1.dev.applepeak.com heartbeat: [17335]: debug: StartNextRemoteRscReq(): child count 1
Sep 15 16:21:45 staging1.dev.applepeak.com heartbeat: [17463]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
harc[17463]:	2011/09/15_16:21:45 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ip-request-resp[17463]:	2011/09/15_16:21:45 received ip-request-resp IPaddr::10.10.10.31/255.255.255.0 OK yes
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:45 info: Acquiring resource group: staging1.dev.applepeak.com IPaddr::10.10.10.31/255.255.255.0 mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[17517]:	2011/09/15_16:21:46 INFO:  Resource is stopped
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:46 info: Running /etc/ha.d/resource.d/IPaddr 10.10.10.31/255.255.255.0 start
Converted dotted-quad netmask to CIDR as: 24
/usr/lib/ocf/resource.d/heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[17598]:	2011/09/15_16:21:46 INFO: Using calculated nic for 10.10.10.31: eth0
/usr/lib/ocf/resource.d/heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[17598]:	2011/09/15_16:21:46 INFO: eval ifconfig eth0:0 10.10.10.31 netmask 255.255.255.0 broadcast 10.10.10.255
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[17575]:	2011/09/15_16:21:46 INFO:  Success
INFO:  Success
/etc/ha.d/resource.d/mysql: line 3: This: command not found
/etc/ha.d/resource.d/mysql: line 5: Mar: command not found
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:46 info: Running /etc/ha.d/resource.d/mysql  start
/etc/ha.d/resource.d/mysql: line 3: This: command not found
/etc/ha.d/resource.d/mysql: line 5: Mar: command not found
/etc/ha.d/resource.d/mysql: line 12: Using: command not found
/etc/ha.d/resource.d/mysql: line 13: 17: command not found
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
mysql[17720]:	2011/09/15_16:21:47 Starting mysqld: [ OK ]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:47 ERROR: Return code 127 from /etc/ha.d/resource.d/mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:47 CRIT: Giving up resources due to failure of mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:47 info: Releasing resource group: staging1.dev.applepeak.com IPaddr::10.10.10.31/255.255.255.0 mysql
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:47 info: Running /etc/ha.d/resource.d/mysql  stop
/etc/ha.d/resource.d/mysql: line 3: This: command not found
/etc/ha.d/resource.d/mysql: line 5: Mar: command not found
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
mysql[17925]:	2011/09/15_16:21:52 Stopping mysqld: [ OK ]
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
ResourceManager[17488]:	2011/09/15_16:21:52 info: Running /etc/ha.d/resource.d/IPaddr 10.10.10.31/255.255.255.0 stop
/usr/lib/ocf/resource.d/heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[18024]:	2011/09/15_16:21:52 INFO: ifconfig eth0:0 down
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
IPaddr[18001]:	2011/09/15_16:21:52 INFO:  Success
INFO:  Success
ARPING 10.10.10.31 from 10.10.10.31 eth0
Sent 10 probes (10 broadcast(s))
Received 0 response(s)
/usr/lib/ocf/resource.d//heartbeat/.ocf-shellfuncs: line 182: ha_logger: command not found
hb_standby[18062]:	2011/09/15_16:22:22 Going standby [foreign].
Sep 15 16:22:22 staging1.dev.applepeak.com heartbeat: [17335]: info: staging1.dev.applepeak.com wants to go standby [foreign]
Sep 15 16:22:23 staging1.dev.applepeak.com heartbeat: [17335]: info: standby: staging2.dev.applepeak.com can take our foreign resources
Sep 15 16:22:23 staging1.dev.applepeak.com heartbeat: [18078]: info: give up foreign HA resources (standby).
Sep 15 16:22:23 staging1.dev.applepeak.com heartbeat: [18078]: info: foreign HA resource release completed (standby).
Sep 15 16:22:23 staging1.dev.applepeak.com heartbeat: [17335]: info: Local standby process completed [foreign].
Sep 15 16:22:24 staging1.dev.applepeak.com heartbeat: [17335]: WARN: 1 lost packet(s) for [staging2.dev.applepeak.com] [39:41]
Sep 15 16:22:24 staging1.dev.applepeak.com heartbeat: [17335]: info: remote resource transition completed.
Sep 15 16:22:24 staging1.dev.applepeak.com heartbeat: [17335]: info: No pkts missing from staging2.dev.applepeak.com!
Sep 15 16:22:24 staging1.dev.applepeak.com heartbeat: [17335]: info: Other node completed standby takeover of foreign resources.