Hello, It seems I can't reboot my primary without losing all my services. Primary goes down, secondary picks up and starts, then for some reason immediately stops all resources.
When the primary comes back up, both nodes are in standby mode and I have to issue a hb_takeover on the primary to get resources running again. If the secondary goes down, primary carries on servicing resources as normal. Whilst the primary is down, it is impossible to persuade heartbeat to start resources on the secondary. This is clearly a problem for kernel upgrades, hardware servicing etc. I need to be able to take the primary down on occasion without losing all resources. Using 2.1.4 with drbd 8.3.0 in a 2 node R1 style cluster. Attached: ha.cf, haresources & logfiles from primary & secondary nodes. Regards Stef -- Stefan Morrell | Operations Director Tel: 0845 3452820 | Alpha Omega Computers Ltd Fax: 0845 3452830 | Incorporating Level 5 Internet [email protected] | [email protected] Standard Disclaimer: http://www.aoc-uk.com/16.asp Alpha Omega Computers Ltd, Unit 57, BBTC, Grange Road, Batley, WF17 6ER. Registered in England No. 3867142. VAT No. GB734421454
haresources
Description: haresources
ha.cf
Description: ha.cf
Feb 24 12:28:33 fedecks-1 heartbeat: [961]: info: Heartbeat shutdown in progress. (961) Feb 24 12:28:33 fedecks-1 heartbeat: [32429]: info: Giving up all HA resources. Feb 24 12:28:33 fedecks-1 ResourceManager[32442]: [32453]: info: Releasing resource group: fedecks-1 drbddisk::shared Filesystem::/dev/drbd0::/shared::ext3 drbdlinks 194.168.159.3 postgrey postfix clamd MailScanner Feb 24 12:28:33 fedecks-1 ResourceManager[32442]: [32463]: info: Running /etc/ha.d/resource.d/MailScanner stop Feb 24 12:28:33 fedecks-1 ResourceManager[32442]: [32464]: debug: Starting /etc/ha.d/resource.d/MailScanner stop Feb 24 12:28:33 fedecks-1 ResourceManager[32442]: [32467]: debug: /etc/ha.d/resource.d/MailScanner stop done. RC=0 Feb 24 12:28:34 fedecks-1 ResourceManager[32442]: [32476]: info: Running /etc/ha.d/resource.d/clamd stop Feb 24 12:28:34 fedecks-1 ResourceManager[32442]: [32477]: debug: Starting /etc/ha.d/resource.d/clamd stop Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32489]: debug: /etc/ha.d/resource.d/clamd stop done. RC=0 Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32498]: info: Running /etc/ha.d/resource.d/postfix stop Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32499]: debug: Starting /etc/ha.d/resource.d/postfix stop Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32509]: debug: /etc/ha.d/resource.d/postfix stop done. RC=0 Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32518]: info: Running /etc/ha.d/resource.d/postgrey stop Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32519]: debug: Starting /etc/ha.d/resource.d/postgrey stop Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32522]: debug: /etc/ha.d/resource.d/postgrey stop done. RC=0 Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32538]: info: Running /etc/ha.d/resource.d/IPaddr 194.168.159.3 stop Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32539]: debug: Starting /etc/ha.d/resource.d/IPaddr 194.168.159.3 stop Feb 24 12:28:36 fedecks-1 IPaddr[32557]: [32572]: INFO: ifconfig eth1:0 down Feb 24 12:28:36 fedecks-1 IPaddr[32540]: [32575]: INFO: Success Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32576]: debug: /etc/ha.d/resource.d/IPaddr 194.168.159.3 stop done. RC=0 Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32585]: info: Running /etc/ha.d/resource.d/drbdlinks stop Feb 24 12:28:36 fedecks-1 ResourceManager[32442]: [32586]: debug: Starting /etc/ha.d/resource.d/drbdlinks stop Feb 24 12:28:40 fedecks-1 ResourceManager[32442]: [32614]: debug: /etc/ha.d/resource.d/drbdlinks stop done. RC=1 Feb 24 12:28:40 fedecks-1 ResourceManager[32442]: [32615]: ERROR: Return code 1 from /etc/ha.d/resource.d/drbdlinks Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32618]: info: Retrying failed stop operation [drbdlinks] Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32627]: info: Running /etc/ha.d/resource.d/drbdlinks stop Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32628]: debug: Starting /etc/ha.d/resource.d/drbdlinks stop Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32630]: debug: /etc/ha.d/resource.d/drbdlinks stop done. RC=0 Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32644]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 stop Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32645]: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 stop Feb 24 12:28:41 fedecks-1 Filesystem[32657]: [32687]: INFO: Running stop for /dev/drbd0 on /shared Feb 24 12:28:41 fedecks-1 Filesystem[32657]: [32697]: INFO: Trying to unmount /shared Feb 24 12:28:41 fedecks-1 Filesystem[32657]: [32699]: INFO: unmounted /shared successfully Feb 24 12:28:41 fedecks-1 Filesystem[32646]: [32705]: INFO: Success Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32706]: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 stop done. RC=0 Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32720]: info: Running /etc/ha.d/resource.d/drbddisk shared stop Feb 24 12:28:41 fedecks-1 ResourceManager[32442]: [32721]: debug: Starting /etc/ha.d/resource.d/drbddisk shared stop Feb 24 12:28:42 fedecks-1 ResourceManager[32442]: [32726]: debug: /etc/ha.d/resource.d/drbddisk shared stop done. RC=0 Feb 24 12:28:42 fedecks-1 heartbeat: [32429]: info: All HA resources relinquished. Feb 24 12:28:43 fedecks-1 heartbeat: [961]: info: killing /usr/lib/heartbeat/ipfail process group 1007 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: killing HBREAD process 992 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: killing HBFIFO process 986 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: killing HBWRITE process 987 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: killing HBREAD process 988 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: killing HBWRITE process 989 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: killing HBREAD process 990 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: killing HBWRITE process 991 with signal 15 Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: Core process 986 exited. 7 remaining Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: Core process 987 exited. 6 remaining Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: Core process 988 exited. 5 remaining Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: Core process 989 exited. 4 remaining Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: Core process 990 exited. 3 remaining Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: Core process 991 exited. 2 remaining Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: Core process 992 exited. 1 remaining Feb 24 12:28:44 fedecks-1 heartbeat: [961]: info: fedecks-1 Heartbeat shutdown complete. Feb 24 12:28:45 fedecks-1 logd: [911]: debug: logd_term_action: received SIGTERM Feb 24 12:28:45 fedecks-1 logd: [911]: debug: logd_term_action: waiting for 0 messages to be read by write process Feb 24 12:28:45 fedecks-1 logd: [911]: debug: logd_term_action: sending SIGTERM to write process Feb 24 12:28:45 fedecks-1 logd: [920]: info: logd_term_write_action: received SIGTERM Feb 24 12:28:45 fedecks-1 logd: [920]: debug: Writing out 0 messages then quitting Feb 24 12:28:45 fedecks-1 logd: [920]: info: Exiting write process Feb 24 12:30:34 fedecks-1 logd: [910]: info: logd started with /etc/logd.cf. Feb 24 12:30:34 fedecks-1 logd: [910]: WARN: Core dumps could be lost if multiple dumps occur. Feb 24 12:30:34 fedecks-1 logd: [910]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability Feb 24 12:30:34 fedecks-1 logd: [910]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability Feb 24 12:30:34 fedecks-1 logd: [910]: info: G_main_add_SignalHandler: Added signal handler for signal 15 Feb 24 12:30:34 fedecks-1 logd: [918]: info: G_main_add_SignalHandler: Added signal handler for signal 15 Feb 24 12:30:35 fedecks-1 heartbeat: [959]: info: No log entry found in ha.cf -- use logd Feb 24 12:30:35 fedecks-1 heartbeat: [959]: info: Enabling logging daemon Feb 24 12:30:35 fedecks-1 heartbeat: [959]: info: logfile and debug file are those specified in logd config file (default /etc/logd.cf) Feb 24 12:30:35 fedecks-1 heartbeat: [959]: info: ************************** Feb 24 12:30:35 fedecks-1 heartbeat: [959]: info: Configuration validated. Starting heartbeat 2.1.4 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: heartbeat: version 2.1.4 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: Heartbeat generation: 1233854551 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth2 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth2 - Status: 1 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: glib: ping group heartbeat started. Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: G_main_add_TriggerHandler: Added signal manual handler Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: G_main_add_TriggerHandler: Added signal manual handler Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Feb 24 12:30:35 fedecks-1 heartbeat: [960]: info: Local status now set to: 'up' Feb 24 12:30:36 fedecks-1 heartbeat: [960]: info: Link fedecks-2:eth0 up. Feb 24 12:30:36 fedecks-1 heartbeat: [960]: info: Status update for node fedecks-2: status active Feb 24 12:30:36 fedecks-1 heartbeat: [993]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Feb 24 12:30:36 fedecks-1 heartbeat: [960]: info: Link fedecks-2:eth2 up. Feb 24 12:30:36 fedecks-1 heartbeat: [960]: info: Link external:external up. Feb 24 12:30:36 fedecks-1 heartbeat: [960]: info: Status update for node external: status ping Feb 24 12:30:36 fedecks-1 heartbeat: [960]: info: Link fedecks-1:eth0 up. Feb 24 12:30:36 fedecks-1 heartbeat: [960]: info: Link fedecks-1:eth2 up. Feb 24 12:30:36 fedecks-1 harc[993]: [1000]: info: Running /etc/ha.d/rc.d/status status Feb 24 12:30:37 fedecks-1 heartbeat: [960]: info: Comm_now_up(): updating status to active Feb 24 12:30:37 fedecks-1 heartbeat: [960]: info: Local status now set to: 'active' Feb 24 12:30:37 fedecks-1 heartbeat: [960]: info: Starting child client "/usr/lib/heartbeat/ipfail" (17,65) Feb 24 12:30:37 fedecks-1 heartbeat: [1006]: info: Starting "/usr/lib/heartbeat/ipfail" as uid 17 gid 65 (pid 1006) Feb 24 12:30:37 fedecks-1 ipfail: [1006]: debug: PID=1006 Feb 24 12:30:37 fedecks-1 ipfail: [1006]: debug: Signing in with heartbeat Feb 24 12:30:37 fedecks-1 heartbeat: [960]: info: remote resource transition completed. Feb 24 12:30:37 fedecks-1 ipfail: [1006]: debug: [We are fedecks-1] Feb 24 12:30:37 fedecks-1 heartbeat: [960]: info: remote resource transition completed. Feb 24 12:30:37 fedecks-1 heartbeat: [960]: info: Local Resource acquisition completed. (none) Feb 24 12:30:37 fedecks-1 heartbeat: [960]: info: Initial resource acquisition complete (T_RESOURCES(them)) Feb 24 12:30:37 fedecks-1 ipfail: [1006]: debug: auto_failback -> 0 (off) Feb 24 12:30:38 fedecks-1 ipfail: [1006]: debug: Setting message filter mode Feb 24 12:30:38 fedecks-1 ipfail: [1006]: debug: Starting node walk Feb 24 12:30:39 fedecks-1 ipfail: [1006]: debug: Cluster node: external: status: ping Feb 24 12:30:40 fedecks-1 ipfail: [1006]: debug: Cluster node: fedecks-2: status: active Feb 24 12:30:40 fedecks-1 ipfail: [1006]: debug: [They are fedecks-2] Feb 24 12:30:40 fedecks-1 ipfail: [1006]: debug: Cluster node: fedecks-1: status: active Feb 24 12:30:41 fedecks-1 ipfail: [1006]: debug: Setting message signal Feb 24 12:30:41 fedecks-1 ipfail: [1006]: debug: Waiting for messages... Feb 24 12:30:42 fedecks-1 ipfail: [1006]: debug: Other side is now stable. Feb 24 12:30:42 fedecks-1 ipfail: [1006]: debug: Other side is now stable. Feb 24 12:30:42 fedecks-1 ipfail: [1006]: debug: Got asked for num_ping. Feb 24 12:30:43 fedecks-1 ipfail: [1006]: debug: Found ping node external! Feb 24 12:30:43 fedecks-1 ipfail: [1006]: info: Ping node count is balanced. Feb 24 12:30:43 fedecks-1 ipfail: [1006]: debug: Abort message sent. Feb 24 12:33:28 fedecks-1 heartbeat: [1032]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Feb 24 12:33:28 fedecks-1 harc[1032]: [1038]: info: Running /etc/ha.d/rc.d/hb_takeover hb_takeover Feb 24 12:33:28 fedecks-1 heartbeat: [960]: info: fedecks-2 wants to go standby [all] Feb 24 12:33:29 fedecks-1 ipfail: [1006]: debug: Other side is unstable. Feb 24 12:33:55 fedecks-1 heartbeat: [960]: info: standby: acquire [all] resources from fedecks-2 Feb 24 12:33:55 fedecks-1 heartbeat: [1050]: info: acquire all HA resources (standby). Feb 24 12:33:55 fedecks-1 ResourceManager[1063]: [1074]: info: Acquiring resource group: fedecks-1 drbddisk::shared Filesystem::/dev/drbd0::/shared::ext3 drbdlinks 194.168.159.3 postgrey postfix clamd MailScanner Feb 24 12:33:55 fedecks-1 ResourceManager[1063]: [1103]: info: Running /etc/ha.d/resource.d/drbddisk shared start Feb 24 12:33:55 fedecks-1 ResourceManager[1063]: [1104]: debug: Starting /etc/ha.d/resource.d/drbddisk shared start Feb 24 12:33:55 fedecks-1 ResourceManager[1063]: [1109]: debug: /etc/ha.d/resource.d/drbddisk shared start done. RC=0 Feb 24 12:33:55 fedecks-1 Filesystem[1121]: [1165]: INFO: Resource is stopped Feb 24 12:33:55 fedecks-1 ResourceManager[1063]: [1179]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 start Feb 24 12:33:55 fedecks-1 ResourceManager[1063]: [1180]: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 start Feb 24 12:33:56 fedecks-1 Filesystem[1192]: [1222]: INFO: Running start for /dev/drbd0 on /shared Feb 24 12:33:56 fedecks-1 Filesystem[1181]: [1236]: INFO: Success Feb 24 12:33:56 fedecks-1 ResourceManager[1063]: [1237]: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 start done. RC=0 Feb 24 12:33:56 fedecks-1 ResourceManager[1063]: [1253]: info: Running /etc/ha.d/resource.d/drbdlinks start Feb 24 12:33:56 fedecks-1 ResourceManager[1063]: [1254]: debug: Starting /etc/ha.d/resource.d/drbdlinks start Feb 24 12:34:00 fedecks-1 ResourceManager[1063]: [1299]: debug: /etc/ha.d/resource.d/drbdlinks start done. RC=0 Feb 24 12:34:00 fedecks-1 IPaddr[1311]: [1342]: INFO: Resource is stopped Feb 24 12:34:01 fedecks-1 ResourceManager[1063]: [1358]: info: Running /etc/ha.d/resource.d/IPaddr 194.168.159.3 start Feb 24 12:34:01 fedecks-1 ResourceManager[1063]: [1359]: debug: Starting /etc/ha.d/resource.d/IPaddr 194.168.159.3 start Feb 24 12:34:01 fedecks-1 IPaddr[1377]: [1408]: INFO: Using calculated nic for 194.168.159.3: eth1 Feb 24 12:34:01 fedecks-1 IPaddr[1377]: [1413]: INFO: Using calculated netmask for 194.168.159.3: 255.255.255.0 Feb 24 12:34:01 fedecks-1 IPaddr[1377]: [1418]: DEBUG: Using calculated broadcast for 194.168.159.3: 194.168.159.255 Feb 24 12:34:01 fedecks-1 IPaddr[1377]: [1435]: INFO: eval ifconfig eth1:0 194.168.159.3 netmask 255.255.255.0 broadcast 194.168.159.255 Feb 24 12:34:01 fedecks-1 IPaddr[1377]: [1440]: DEBUG: Sending Gratuitous Arp for 194.168.159.3 on eth1:0 [eth1] Feb 24 12:34:01 fedecks-1 IPaddr[1360]: [1454]: INFO: Success Feb 24 12:34:01 fedecks-1 ResourceManager[1063]: [1455]: debug: /etc/ha.d/resource.d/IPaddr 194.168.159.3 start done. RC=0 Feb 24 12:34:01 fedecks-1 ResourceManager[1063]: [1471]: info: Running /etc/ha.d/resource.d/postgrey start Feb 24 12:34:01 fedecks-1 ResourceManager[1063]: [1472]: debug: Starting /etc/ha.d/resource.d/postgrey start Feb 24 12:34:02 fedecks-1 ResourceManager[1063]: [1476]: debug: /etc/ha.d/resource.d/postgrey start done. RC=0 Feb 24 12:34:02 fedecks-1 ResourceManager[1063]: [1494]: info: Running /etc/ha.d/resource.d/postfix start Feb 24 12:34:02 fedecks-1 ResourceManager[1063]: [1495]: debug: Starting /etc/ha.d/resource.d/postfix start Feb 24 12:34:03 fedecks-1 ResourceManager[1063]: [1575]: debug: /etc/ha.d/resource.d/postfix start done. RC=0 Feb 24 12:34:03 fedecks-1 ResourceManager[1063]: [1597]: info: Running /etc/ha.d/resource.d/clamd start Feb 24 12:34:03 fedecks-1 ResourceManager[1063]: [1598]: debug: Starting /etc/ha.d/resource.d/clamd start Feb 24 12:34:08 fedecks-1 ResourceManager[1063]: [1677]: debug: /etc/ha.d/resource.d/clamd start done. RC=0 Feb 24 12:34:08 fedecks-1 ResourceManager[1063]: [1693]: info: Running /etc/ha.d/resource.d/MailScanner start Feb 24 12:34:08 fedecks-1 ResourceManager[1063]: [1694]: debug: Starting /etc/ha.d/resource.d/MailScanner start Feb 24 12:34:12 fedecks-1 ResourceManager[1063]: [1738]: debug: /etc/ha.d/resource.d/MailScanner start done. RC=0 Feb 24 12:34:12 fedecks-1 heartbeat: [1050]: info: all HA resource acquisition completed (standby). Feb 24 12:34:12 fedecks-1 heartbeat: [960]: info: Standby resource acquisition done [all]. Feb 24 12:34:13 fedecks-1 heartbeat: [960]: info: remote resource transition completed. Feb 24 12:34:13 fedecks-1 ipfail: [1006]: debug: Other side is now stable. Feb 24 12:34:13 fedecks-1 ipfail: [1006]: debug: Other side is now stable.
Feb 24 12:28:33 fedecks-2 ipfail: [1006]: debug: Other side is unstable. Feb 24 12:28:42 fedecks-2 heartbeat: [1025]: info: acquire all HA resources (standby). Feb 24 12:28:42 fedecks-2 heartbeat: [960]: info: Received shutdown notice from 'fedecks-1'. Feb 24 12:28:42 fedecks-2 heartbeat: [960]: info: Resources being acquired from fedecks-1. Feb 24 12:28:42 fedecks-2 heartbeat: [960]: debug: StartNextRemoteRscReq(): child count 1 Feb 24 12:28:42 fedecks-2 heartbeat: [1027]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys fedecks-2] to acquire. Feb 24 12:28:42 fedecks-2 heartbeat: [960]: debug: StartNextRemoteRscReq(): child count 1 Feb 24 12:28:42 fedecks-2 ResourceManager[1045]: [1062]: info: Acquiring resource group: fedecks-1 drbddisk::shared Filesystem::/dev/drbd0::/shared::ext3 drbdlinks 194.168.159.3 postgrey postfix clamd MailScanner Feb 24 12:28:42 fedecks-2 ResourceManager[1045]: [1091]: info: Running /etc/ha.d/resource.d/drbddisk shared start Feb 24 12:28:42 fedecks-2 ResourceManager[1045]: [1092]: debug: Starting /etc/ha.d/resource.d/drbddisk shared start Feb 24 12:28:42 fedecks-2 ResourceManager[1045]: [1097]: debug: /etc/ha.d/resource.d/drbddisk shared start done. RC=0 Feb 24 12:28:42 fedecks-2 Filesystem[1109]: [1153]: INFO: Resource is stopped Feb 24 12:28:42 fedecks-2 ResourceManager[1045]: [1167]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 start Feb 24 12:28:42 fedecks-2 ResourceManager[1045]: [1168]: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 start Feb 24 12:28:42 fedecks-2 Filesystem[1180]: [1210]: INFO: Running start for /dev/drbd0 on /shared Feb 24 12:28:42 fedecks-2 Filesystem[1169]: [1224]: INFO: Success Feb 24 12:28:42 fedecks-2 ResourceManager[1045]: [1225]: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 start done. RC=0 Feb 24 12:28:43 fedecks-2 ResourceManager[1045]: [1241]: info: Running /etc/ha.d/resource.d/drbdlinks start Feb 24 12:28:43 fedecks-2 ResourceManager[1045]: [1242]: debug: Starting /etc/ha.d/resource.d/drbdlinks start Feb 24 12:28:47 fedecks-2 ResourceManager[1045]: [1287]: debug: /etc/ha.d/resource.d/drbdlinks start done. RC=0 Feb 24 12:28:47 fedecks-2 IPaddr[1299]: [1330]: INFO: Resource is stopped Feb 24 12:28:47 fedecks-2 ResourceManager[1045]: [1346]: info: Running /etc/ha.d/resource.d/IPaddr 194.168.159.3 start Feb 24 12:28:47 fedecks-2 ResourceManager[1045]: [1347]: debug: Starting /etc/ha.d/resource.d/IPaddr 194.168.159.3 start Feb 24 12:28:47 fedecks-2 IPaddr[1365]: [1396]: INFO: Using calculated nic for 194.168.159.3: eth1 Feb 24 12:28:47 fedecks-2 IPaddr[1365]: [1401]: INFO: Using calculated netmask for 194.168.159.3: 255.255.255.0 Feb 24 12:28:47 fedecks-2 IPaddr[1365]: [1406]: DEBUG: Using calculated broadcast for 194.168.159.3: 194.168.159.255 Feb 24 12:28:47 fedecks-2 IPaddr[1365]: [1423]: INFO: eval ifconfig eth1:0 194.168.159.3 netmask 255.255.255.0 broadcast 194.168.159.255 Feb 24 12:28:48 fedecks-2 IPaddr[1365]: [1428]: DEBUG: Sending Gratuitous Arp for 194.168.159.3 on eth1:0 [eth1] Feb 24 12:28:48 fedecks-2 IPaddr[1348]: [1442]: INFO: Success Feb 24 12:28:48 fedecks-2 ResourceManager[1045]: [1443]: debug: /etc/ha.d/resource.d/IPaddr 194.168.159.3 start done. RC=0 Feb 24 12:28:48 fedecks-2 ResourceManager[1045]: [1459]: info: Running /etc/ha.d/resource.d/postgrey start Feb 24 12:28:48 fedecks-2 ResourceManager[1045]: [1460]: debug: Starting /etc/ha.d/resource.d/postgrey start Feb 24 12:28:48 fedecks-2 ResourceManager[1045]: [1464]: debug: /etc/ha.d/resource.d/postgrey start done. RC=0 Feb 24 12:28:48 fedecks-2 ResourceManager[1045]: [1482]: info: Running /etc/ha.d/resource.d/postfix start Feb 24 12:28:48 fedecks-2 ResourceManager[1045]: [1483]: debug: Starting /etc/ha.d/resource.d/postfix start Feb 24 12:28:50 fedecks-2 ResourceManager[1045]: [1563]: debug: /etc/ha.d/resource.d/postfix start done. RC=0 Feb 24 12:28:50 fedecks-2 ResourceManager[1045]: [1584]: info: Running /etc/ha.d/resource.d/clamd start Feb 24 12:28:50 fedecks-2 ResourceManager[1045]: [1585]: debug: Starting /etc/ha.d/resource.d/clamd start Feb 24 12:28:55 fedecks-2 ResourceManager[1045]: [1623]: debug: /etc/ha.d/resource.d/clamd start done. RC=0 Feb 24 12:28:55 fedecks-2 ResourceManager[1045]: [1639]: info: Running /etc/ha.d/resource.d/MailScanner start Feb 24 12:28:55 fedecks-2 ResourceManager[1045]: [1640]: debug: Starting /etc/ha.d/resource.d/MailScanner start Feb 24 12:28:59 fedecks-2 ResourceManager[1045]: [1660]: debug: /etc/ha.d/resource.d/MailScanner start done. RC=0 Feb 24 12:28:59 fedecks-2 heartbeat: [1025]: info: all HA resource acquisition completed (standby). Feb 24 12:28:59 fedecks-2 heartbeat: [960]: info: Standby resource acquisition done [all]. Feb 24 12:28:59 fedecks-2 heartbeat: [1661]: debug: notify_world: setting SIGCHLD Handler to SIG_DFL Feb 24 12:28:59 fedecks-2 harc[1661]: [1667]: info: Running /etc/ha.d/rc.d/status status Feb 24 12:28:59 fedecks-2 mach_down[1673]: [1694]: info: Taking over resource group drbddisk::shared Feb 24 12:28:59 fedecks-2 ResourceManager[1695]: [1706]: info: Acquiring resource group: fedecks-1 drbddisk::shared Filesystem::/dev/drbd0::/shared::ext3 drbdlinks 194.168.159.3 postgrey postfix clamd MailScanner Feb 24 12:29:00 fedecks-2 Filesystem[1733]: [1777]: INFO: Running OK Feb 24 12:29:00 fedecks-2 IPaddr[1796]: [1827]: INFO: Running OK Feb 24 12:29:00 fedecks-2 ResourceManager[1695]: [1843]: info: Running /etc/ha.d/resource.d/postgrey start Feb 24 12:29:00 fedecks-2 heartbeat: [960]: WARN: node fedecks-1: is dead Feb 24 12:29:00 fedecks-2 ipfail: [1006]: info: Status update: Node fedecks-1 now has status dead Feb 24 12:29:00 fedecks-2 heartbeat: [960]: info: Dead node fedecks-1 gave up resources. Feb 24 12:29:00 fedecks-2 heartbeat: [960]: info: Link fedecks-1:eth0 dead. Feb 24 12:29:00 fedecks-2 heartbeat: [960]: info: Link fedecks-1:eth2 dead. Feb 24 12:29:00 fedecks-2 ResourceManager[1695]: [1844]: debug: Starting /etc/ha.d/resource.d/postgrey start Feb 24 12:29:00 fedecks-2 ResourceManager[1695]: [1847]: debug: /etc/ha.d/resource.d/postgrey start done. RC=9 Feb 24 12:29:00 fedecks-2 ResourceManager[1695]: [1848]: ERROR: Return code 9 from /etc/ha.d/resource.d/postgrey Feb 24 12:29:00 fedecks-2 ResourceManager[1695]: [1849]: CRIT: Giving up resources due to failure of postgrey Feb 24 12:29:00 fedecks-2 ResourceManager[1695]: [1850]: info: Releasing resource group: fedecks-1 drbddisk::shared Filesystem::/dev/drbd0::/shared::ext3 drbdlinks 194.168.159.3 postgrey postfix clamd MailScanner Feb 24 12:29:01 fedecks-2 ResourceManager[1695]: [1860]: info: Running /etc/ha.d/resource.d/MailScanner stop Feb 24 12:29:01 fedecks-2 ResourceManager[1695]: [1861]: debug: Starting /etc/ha.d/resource.d/MailScanner stop Feb 24 12:29:01 fedecks-2 ResourceManager[1695]: [1864]: debug: /etc/ha.d/resource.d/MailScanner stop done. RC=0 Feb 24 12:29:01 fedecks-2 ResourceManager[1695]: [1873]: info: Running /etc/ha.d/resource.d/clamd stop Feb 24 12:29:01 fedecks-2 ResourceManager[1695]: [1874]: debug: Starting /etc/ha.d/resource.d/clamd stop Feb 24 12:29:01 fedecks-2 ipfail: [1006]: debug: Found ping node external! Feb 24 12:29:01 fedecks-2 ipfail: [1006]: info: NS: We are still alive! Feb 24 12:29:01 fedecks-2 ipfail: [1006]: info: Link Status update: Link fedecks-1/eth0 now has status dead Feb 24 12:29:02 fedecks-2 ipfail: [1006]: debug: Found ping node external! Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1884]: debug: /etc/ha.d/resource.d/clamd stop done. RC=0 Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1893]: info: Running /etc/ha.d/resource.d/postfix stop Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1894]: debug: Starting /etc/ha.d/resource.d/postfix stop Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1904]: debug: /etc/ha.d/resource.d/postfix stop done. RC=0 Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1913]: info: Running /etc/ha.d/resource.d/postgrey stop Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1914]: debug: Starting /etc/ha.d/resource.d/postgrey stop Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1917]: debug: /etc/ha.d/resource.d/postgrey stop done. RC=0 Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1933]: info: Running /etc/ha.d/resource.d/IPaddr 194.168.159.3 stop Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1934]: debug: Starting /etc/ha.d/resource.d/IPaddr 194.168.159.3 stop Feb 24 12:29:03 fedecks-2 IPaddr[1952]: [1967]: INFO: ifconfig eth1:0 down Feb 24 12:29:03 fedecks-2 IPaddr[1935]: [1970]: INFO: Success Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1971]: debug: /etc/ha.d/resource.d/IPaddr 194.168.159.3 stop done. RC=0 Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1980]: info: Running /etc/ha.d/resource.d/drbdlinks stop Feb 24 12:29:03 fedecks-2 ResourceManager[1695]: [1981]: debug: Starting /etc/ha.d/resource.d/drbdlinks stop Feb 24 12:29:03 fedecks-2 ipfail: [1006]: info: Asking other side for ping node count. Feb 24 12:29:03 fedecks-2 ipfail: [1006]: debug: Message [num_ping] sent. Feb 24 12:29:03 fedecks-2 ipfail: [1006]: info: Checking remote count of ping nodes. Feb 24 12:29:03 fedecks-2 ipfail: [1006]: info: Link Status update: Link fedecks-1/eth2 now has status dead Feb 24 12:29:04 fedecks-2 ipfail: [1006]: debug: Found ping node external! Feb 24 12:29:07 fedecks-2 ResourceManager[1695]: [2026]: debug: /etc/ha.d/resource.d/drbdlinks stop done. RC=0 Feb 24 12:29:07 fedecks-2 ResourceManager[1695]: [2040]: info: Running /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 stop Feb 24 12:29:07 fedecks-2 ResourceManager[1695]: [2041]: debug: Starting /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 stop Feb 24 12:29:07 fedecks-2 Filesystem[2053]: [2083]: INFO: Running stop for /dev/drbd0 on /shared Feb 24 12:29:07 fedecks-2 Filesystem[2053]: [2093]: INFO: Trying to unmount /shared Feb 24 12:29:08 fedecks-2 Filesystem[2053]: [2095]: INFO: unmounted /shared successfully Feb 24 12:29:08 fedecks-2 Filesystem[2042]: [2101]: INFO: Success Feb 24 12:29:08 fedecks-2 ResourceManager[1695]: [2102]: debug: /etc/ha.d/resource.d/Filesystem /dev/drbd0 /shared ext3 stop done. RC=0 Feb 24 12:29:08 fedecks-2 ResourceManager[1695]: [2116]: info: Running /etc/ha.d/resource.d/drbddisk shared stop Feb 24 12:29:08 fedecks-2 ResourceManager[1695]: [2117]: debug: Starting /etc/ha.d/resource.d/drbddisk shared stop Feb 24 12:29:08 fedecks-2 ResourceManager[1695]: [2122]: debug: /etc/ha.d/resource.d/drbddisk shared stop done. RC=0 Feb 24 12:29:08 fedecks-2 mach_down[1673]: [2125]: info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired Feb 24 12:29:08 fedecks-2 mach_down[1673]: [2129]: info: mach_down takeover complete for node fedecks-1. Feb 24 12:29:08 fedecks-2 heartbeat: [960]: info: mach_down takeover complete. Feb 24 12:29:38 fedecks-2 hb_standby[2139]: [2145]: Going standby [foreign]. Feb 24 12:29:38 fedecks-2 heartbeat: [960]: info: fedecks-2 wants to go standby [foreign] Feb 24 12:29:49 fedecks-2 heartbeat: [960]: WARN: No reply to standby request. Standby request cancelled.
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
