There are two nodes in the cluster, and I shutdown haproxy1, and start a new node and name it as haproxy1 and set its uuid to the same as the dead haproxy1. Here's the log from haproxy2, the ever living node: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cib[2924]: 2009/07/24_10:55:25 info: cib_ccm_msg_callback: LOST: haproxy1.b.cel.brightcove.com ccm[2923]: 2009/07/24_10:55:25 info: Break tie for 2 nodes cluster cib[2924]: 2009/07/24_10:55:25 info: cib_ccm_msg_callback: PEER: haproxy2.b.cel.brightcove.com crmd[2928]: 2009/07/24_10:55:25 info: ccm_event_detail: NEW MEMBERSHIP: trans=3, nodes=1, new=0, lost=1 n_idx=0, new_idx=1, old_idx=3 crmd[2928]: 2009/07/24_10:55:25 info: ccm_event_detail: CURRENT: haproxy2.b.cel.brightcove.com [nodeid=1, born=3] crmd[2928]: 2009/07/24_10:55:25 info: ccm_event_detail: LOST: haproxy1.b.cel.brightcove.com [nodeid=0, born=2] crmd[2928]: 2009/07/24_10:55:25 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] heartbeat[2869]: 2009/07/24_10:55:56 WARN: node haproxy1.b.cel.brightcove.com: is dead heartbeat[2869]: 2009/07/24_10:55:56 info: Link haproxy1.b.cel.brightcove.com:eth0 dead. crmd[2928]: 2009/07/24_10:55:56 notice: crmd_ha_status_callback: Status update: Node haproxy1.b.cel.brightcove.com now has status [dead] logd[2847]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). logd[2853]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). ccm[2923]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). heartbeat[2869]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). attrd[2927]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). stonithd[2926]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). crmd[2928]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). mgmtd[2929]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). tengine[2988]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). pengine[2989]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). heartbeat[2876]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). lrmd[2925]: 2009/07/24_10:56:13 info: time_longclock: clock_t wrapped around (uptime). heartbeat[2877]: 2009/07/24_10:56:14 info: time_longclock: clock_t wrapped around (uptime). heartbeat[2879]: 2009/07/24_10:56:14 info: time_longclock: clock_t wrapped around (uptime). cib[2924]: 2009/07/24_10:56:14 info: time_longclock: clock_t wrapped around (uptime). cib[2924]: 2009/07/24_11:03:40 info: cib_stats: Processed 71 operations (704.00us average, 0% utilization) in the last 10min heartbeat[2880]: 2009/07/24_11:04:18 info: time_longclock: clock_t wrapped around (uptime). heartbeat[2869]: 2009/07/24_11:04:18 info: Heartbeat restart on node haproxy1.b.cel.brightcove.com heartbeat[2869]: 2009/07/24_11:04:18 info: Link haproxy1.b.cel.brightcove.com:eth0 up. heartbeat[2869]: 2009/07/24_11:04:18 info: Status update for node haproxy1.b.cel.brightcove.com: status init heartbeat[2869]: 2009/07/24_11:04:18 info: Status update for node haproxy1.b.cel.brightcove.com: status up crmd[2928]: 2009/07/24_11:04:18 notice: crmd_ha_status_callback: Status update: Node haproxy1.b.cel.brightcove.com now has status [init] crmd[2928]: 2009/07/24_11:04:18 notice: crmd_ha_status_callback: Status update: Node haproxy1.b.cel.brightcove.com now has status [up] heartbeat[2869]: 2009/07/24_11:04:18 info: all clients are now paused heartbeat[2869]: 2009/07/24_11:04:47 info: Status update for node haproxy1.b.cel.brightcove.com: status active crmd[2928]: 2009/07/24_11:04:47 notice: crmd_ha_status_callback: Status update: Node haproxy1.b.cel.brightcove.com now has status [active] cib[2924]: 2009/07/24_11:04:47 info: cib_client_status_callback: Status update: Client haproxy1.b.cel.brightcove.com/cib now has status [join] heartbeat[2869]: 2009/07/24_11:04:51 WARN: 1 lost packet(s) for [ haproxy1.b.cel.brightcove.com] [58:60] heartbeat[2869]: 2009/07/24_11:04:51 info: No pkts missing from haproxy1.b.cel.brightcove.com! crmd[2928]: 2009/07/24_11:04:51 notice: crmd_client_status_callback: Status update: Client haproxy1.b.cel.brightcove.com/crmd now has status [online] heartbeat[2869]: 2009/07/24_11:04:52 WARN: 1 lost packet(s) for [ haproxy1.b.cel.brightcove.com] [62:64] heartbeat[2869]: 2009/07/24_11:04:52 info: No pkts missing from haproxy1.b.cel.brightcove.com! cib[2924]: 2009/07/24_11:05:26 WARN: cib_peer_callback: Discarding cib_slave_all message (66) from haproxy1.b.cel.brightcove.com: not in our membership cib[2924]: 2009/07/24_11:05:26 WARN: cib_peer_callback: Discarding cib_apply_diff message (68) from haproxy1.b.cel.brightcove.com: not in our membership cib[2924]: 2009/07/24_11:05:26 WARN: cib_peer_callback: Discarding cib_apply_diff message (69) from haproxy1.b.cel.brightcove.com: not in our membership cib[2924]: 2009/07/24_11:05:26 WARN: cib_peer_callback: Discarding cib_replace message (6b) from haproxy1.b.cel.brightcove.com: not in our membership cib[2924]: 2009/07/24_11:05:26 WARN: cib_peer_callback: Discarding cib_apply_diff message (6d) from haproxy1.b.cel.brightcove.com: not in our membership cib[2924]: 2009/07/24_11:05:26 WARN: cib_peer_callback: Discarding cib_apply_diff message (6e) from haproxy1.b.cel.brightcove.com: not in our membership cib[2924]: 2009/07/24_11:05:27 WARN: cib_peer_callback: Discarding cib_apply_diff message (71) from haproxy1.b.cel.brightcove.com: not in our membership cib[2924]: 2009/07/24_11:05:27 WARN: cib_peer_callback: Discarding cib_apply_diff message (72) from haproxy1.b.cel.brightcove.com: not in our membership >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Here is log from the new haproxy1: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> logd[3213]: 2009/07/24_11:04:17 info: G_main_add_SignalHandler: Added signal handler for signal 15 logd[3208]: 2009/07/24_11:04:17 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[3229]: 2009/07/24_11:04:17 info: Enabling logging daemon heartbeat[3229]: 2009/07/24_11:04:17 info: logfile and debug file are those specified in logd config file (default /etc/logd.cf) heartbeat[3229]: 2009/07/24_11:04:17 info: Version 2 support: on heartbeat[3229]: 2009/07/24_11:04:17 WARN: logd is enabled but logfile/debugfile/logfacility is still configured in ha.cf heartbeat[3229]: 2009/07/24_11:04:17 info: ************************** heartbeat[3229]: 2009/07/24_11:04:17 info: Configuration validated. Starting heartbeat 2.1.3 heartbeat[3230]: 2009/07/24_11:04:17 info: heartbeat: version 2.1.3 heartbeat[3230]: 2009/07/24_11:04:17 WARN: No Previous generation - starting at 1248447858 heartbeat[3230]: 2009/07/24_11:04:17 info: Heartbeat generation: 1248447858 heartbeat[3230]: 2009/07/24_11:04:17 info: Creating FIFO /var/lib/heartbeat/fifo. heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0 heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: bound send socket to device: eth0 heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: bound receive socket to device: eth0 heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: started on port 694 interface eth0 to 10.254.166.36 heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0 heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: bound send socket to device: eth0 heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: bound receive socket to device: eth0 heartbeat[3230]: 2009/07/24_11:04:17 info: glib: ucast: started on port 694 interface eth0 to 10.255.122.48 heartbeat[3230]: 2009/07/24_11:04:17 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3230]: 2009/07/24_11:04:17 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[3230]: 2009/07/24_11:04:17 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[3230]: 2009/07/24_11:04:17 info: Local status now set to: 'up' heartbeat[3230]: 2009/07/24_11:04:47 WARN: node haproxy2.b.cel.brightcove.com: is dead heartbeat[3230]: 2009/07/24_11:04:47 info: Comm_now_up(): updating status to active heartbeat[3230]: 2009/07/24_11:04:47 info: Local status now set to: 'active' heartbeat[3230]: 2009/07/24_11:04:47 info: Starting child client "/usr/lib/heartbeat/ccm" (498,496) heartbeat[3230]: 2009/07/24_11:04:47 info: Starting child client "/usr/lib/heartbeat/cib" (498,496) heartbeat[3230]: 2009/07/24_11:04:47 info: Starting child client "/usr/lib/heartbeat/lrmd -r" (0,0) heartbeat[3230]: 2009/07/24_11:04:47 info: Starting child client "/usr/lib/heartbeat/stonithd" (0,0) heartbeat[3230]: 2009/07/24_11:04:47 info: Starting child client "/usr/lib/heartbeat/attrd" (498,496) heartbeat[3230]: 2009/07/24_11:04:47 info: Starting child client "/usr/lib/heartbeat/crmd" (498,496) heartbeat[3230]: 2009/07/24_11:04:47 info: Starting child client "/usr/lib/heartbeat/mgmtd -v" (0,0) heartbeat[3265]: 2009/07/24_11:04:47 info: Starting "/usr/lib/heartbeat/ccm" as uid 498 gid 496 (pid 3265) heartbeat[3266]: 2009/07/24_11:04:47 info: Starting "/usr/lib/heartbeat/cib" as uid 498 gid 496 (pid 3266) heartbeat[3267]: 2009/07/24_11:04:47 info: Starting "/usr/lib/heartbeat/lrmd -r" as uid 0 gid 0 (pid 3267) heartbeat[3268]: 2009/07/24_11:04:47 info: Starting "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 3268) heartbeat[3269]: 2009/07/24_11:04:47 info: Starting "/usr/lib/heartbeat/attrd" as uid 498 gid 496 (pid 3269) heartbeat[3270]: 2009/07/24_11:04:47 info: Starting "/usr/lib/heartbeat/crmd" as uid 498 gid 496 (pid 3270) heartbeat[3271]: 2009/07/24_11:04:47 info: Starting "/usr/lib/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 3271) lrmd[3267]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 15 stonithd[3268]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 10 stonithd[3268]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 12 ccm[3265]: 2009/07/24_11:04:47 info: Hostname: haproxy1.b.cel.brightcove.com stonithd[3268]: 2009/07/24_11:04:47 info: Signing in with heartbeat. attrd[3269]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 15 attrd[3269]: 2009/07/24_11:04:47 info: register_with_ha: Hostname: haproxy1.b.cel.brightcove.com stonithd[3268]: 2009/07/24_11:04:47 notice: /usr/lib/heartbeat/stonithd start up successfully. stonithd[3268]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 17 crmd[3270]: 2009/07/24_11:04:47 info: main: CRM Hg Version: node: 552305612591183b1628baa5bc6e903e0f1e26a3 crmd[3270]: 2009/07/24_11:04:47 info: crmd_init: Starting crmd crmd[3270]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 15 crmd[3270]: 2009/07/24_11:04:47 info: G_main_add_TriggerHandler: Added signal manual handler crmd[3270]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 17 lrmd[3267]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 17 lrmd[3267]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 10 lrmd[3267]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 12 lrmd[3267]: 2009/07/24_11:04:47 info: Started. mgmtd[3271]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 15 mgmtd[3271]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 10 mgmtd[3271]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 12 attrd[3269]: 2009/07/24_11:04:47 info: register_with_ha: UUID: 7285647b-01fe-48b3-adc2-b12250b98c96 mgmtd[3271]: 2009/07/24_11:04:47 info: init_crm mgmtd[3271]: 2009/07/24_11:04:47 info: login to cib: 0, ret:-10 cib[3266]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[3266]: 2009/07/24_11:04:47 info: G_main_add_TriggerHandler: Added signal manual handler cib[3266]: 2009/07/24_11:04:47 info: G_main_add_SignalHandler: Added signal handler for signal 17 cib[3266]: 2009/07/24_11:04:47 info: main: Retrieval of a per-action CIB: disabled cib[3266]: 2009/07/24_11:04:47 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3266]: 2009/07/24_11:04:47 WARN: retrieveCib: Cluster configuration not found: /var/lib/heartbeat/crm/cib.xml cib[3266]: 2009/07/24_11:04:47 WARN: readCibXmlFile: Primary configuration corrupt or unusable, trying backup... cib[3266]: 2009/07/24_11:04:47 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3266]: 2009/07/24_11:04:47 WARN: retrieveCib: Cluster configuration not found: /var/lib/heartbeat/crm/cib.xml.last cib[3266]: 2009/07/24_11:04:47 WARN: readCibXmlFile: Continuing with an empty configuration. cib[3266]: 2009/07/24_11:04:47 WARN: readCibXmlFile: No value for admin_epoch was specified in the configuration. cib[3266]: 2009/07/24_11:04:47 WARN: readCibXmlFile: The reccomended course of action is to shutdown, run crm_verify and fix any errors it reports. cib[3266]: 2009/07/24_11:04:47 WARN: readCibXmlFile: We will default to zero and continue but may get confused about which configuration to use if multiple nodes are powered up at the same time. cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] <cib generated="true" admin_epoch="0" epoch="0" num_updates="0" have_quorum="false"> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] <configuration> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] <crm_config/> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] <nodes/> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] <resources/> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] <constraints/> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] </configuration> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] <status/> cib[3266]: 2009/07/24_11:04:47 info: log_data_element: readCibXmlFile: [on-disk] </cib> cib[3266]: 2009/07/24_11:04:47 notice: readCibXmlFile: Enabling DTD validation on the existing (sane) configuration cib[3266]: 2009/07/24_11:04:47 info: startCib: CIB Initialization completed successfully cib[3266]: 2009/07/24_11:04:47 info: cib_register_ha: Signing in with Heartbeat cib[3266]: 2009/07/24_11:04:47 info: cib_register_ha: FSA Hostname: haproxy1.b.cel.brightcove.com cib[3266]: 2009/07/24_11:04:47 info: ccm_connect: Registering with CCM... cib[3266]: 2009/07/24_11:04:47 WARN: ccm_connect: CCM Activation failed cib[3266]: 2009/07/24_11:04:47 WARN: ccm_connect: CCM Connection failed 1 times (30 max) ccm[3265]: 2009/07/24_11:04:50 info: Break tie for 2 nodes cluster ccm[3265]: 2009/07/24_11:04:50 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[3266]: 2009/07/24_11:04:50 info: ccm_connect: Registering with CCM... cib[3266]: 2009/07/24_11:04:51 info: cib_init: Starting cib mainloop cib[3266]: 2009/07/24_11:04:51 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm cib[3266]: 2009/07/24_11:04:51 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 cib[3266]: 2009/07/24_11:04:51 info: cib_ccm_msg_callback: PEER: haproxy1.b.cel.brightcove.com crmd[3270]: 2009/07/24_11:04:51 info: do_cib_control: CIB connection established crmd[3270]: 2009/07/24_11:04:51 info: register_with_ha: Hostname: haproxy1.b.cel.brightcove.com cib[3266]: 2009/07/24_11:04:51 info: cib_client_status_callback: Status update: Client haproxy1.b.cel.brightcove.com/cib now has status [join] cib[3266]: 2009/07/24_11:04:51 info: cib_client_status_callback: Status update: Client haproxy1.b.cel.brightcove.com/cib now has status [online] cib[3266]: 2009/07/24_11:04:51 info: cib_null_callback: Setting cib_refresh_notify callbacks for crmd: on cib[3266]: 2009/07/24_11:04:51 info: cib_null_callback: Setting cib_diff_notify callbacks for mgmtd: on crmd[3270]: 2009/07/24_11:04:51 info: register_with_ha: UUID: 7285647b-01fe-48b3-adc2-b12250b98c96 cib[3272]: 2009/07/24_11:04:52 info: write_cib_contents: Wrote version 0.0.0 of the CIB to disk (digest: 61470affa950eb34c520f6117d4966a5) crmd[3270]: 2009/07/24_11:04:52 info: populate_cib_nodes: Requesting the list of configured nodes cib[3272]: 2009/07/24_11:04:52 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) mgmtd[3271]: 2009/07/24_11:04:52 info: Started. crmd[3270]: 2009/07/24_11:04:53 WARN: get_uuid: Could not calculate UUID for haproxy2.b.cel.brightcove.com crmd[3270]: 2009/07/24_11:04:53 WARN: populate_cib_nodes: Node haproxy2.b.cel.brightcove.com: no uuid found crmd[3270]: 2009/07/24_11:04:53 notice: populate_cib_nodes: Node: haproxy1.b.cel.brightcove.com (uuid: 7285647b-01fe-48b3-adc2-b12250b98c96) crmd[3270]: 2009/07/24_11:04:53 info: do_ha_control: Connected to Heartbeat crmd[3270]: 2009/07/24_11:04:53 info: do_ccm_control: CCM connection established... waiting for first callback crmd[3270]: 2009/07/24_11:04:53 info: do_started: Delaying start, CCM (0000000000100000) not connected crmd[3270]: 2009/07/24_11:04:53 info: crmd_init: Starting crmd's mainloop crmd[3270]: 2009/07/24_11:04:53 notice: crmd_client_status_callback: Status update: Client haproxy1.b.cel.brightcove.com/crmd now has status [online] cib[3273]: 2009/07/24_11:04:53 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3273]: 2009/07/24_11:04:53 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3273]: 2009/07/24_11:04:53 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) cib[3273]: 2009/07/24_11:04:53 info: write_cib_contents: Wrote version 0.0.0 of the CIB to disk (digest: 9b863505d9ee412f3320c664be001f0a) cib[3273]: 2009/07/24_11:04:53 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3273]: 2009/07/24_11:04:53 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3270]: 2009/07/24_11:04:54 notice: crmd_client_status_callback: Status update: Client haproxy1.b.cel.brightcove.com/crmd now has status [online] crmd[3270]: 2009/07/24_11:04:54 info: do_started: Delaying start, CCM (0000000000100000) not connected crmd[3270]: 2009/07/24_11:04:54 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm crmd[3270]: 2009/07/24_11:04:54 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 crmd[3270]: 2009/07/24_11:04:54 info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=1) crmd[3270]: 2009/07/24_11:04:54 info: ccm_event_detail: NEW MEMBERSHIP: trans=1, nodes=1, new=1, lost=0 n_idx=0, new_idx=0, old_idx=3 crmd[3270]: 2009/07/24_11:04:54 info: ccm_event_detail: CURRENT: haproxy1.b.cel.brightcove.com [nodeid=0, born=1] crmd[3270]: 2009/07/24_11:04:54 info: ccm_event_detail: NEW: haproxy1.b.cel.brightcove.com [nodeid=0, born=1] crmd[3270]: 2009/07/24_11:04:54 info: do_started: The local CRM is operational crmd[3270]: 2009/07/24_11:04:54 info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_CCM_CALLBACK origin=do_started ] attrd[3269]: 2009/07/24_11:04:57 info: main: Starting mainloop... crmd[3270]: 2009/07/24_11:05:25 info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped! crmd[3270]: 2009/07/24_11:05:25 WARN: do_log: [[FSA]] Input I_DC_TIMEOUT from crm_timer_popped() received in state (S_PENDING) crmd[3270]: 2009/07/24_11:05:25 info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ] crmd[3270]: 2009/07/24_11:05:25 info: do_election_count_vote: Updated voted hash for haproxy1.b.cel.brightcove.com to vote crmd[3270]: 2009/07/24_11:05:25 info: do_election_count_vote: Election ignore: our vote (haproxy1.b.cel.brightcove.com) crmd[3270]: 2009/07/24_11:05:25 info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] crmd[3270]: 2009/07/24_11:05:25 info: start_subsystem: Starting sub-system "tengine" crmd[3270]: 2009/07/24_11:05:25 info: start_subsystem: Starting sub-system "pengine" cib[3266]: 2009/07/24_11:05:25 info: cib_process_readwrite: We are now in R/W mode crmd[3270]: 2009/07/24_11:05:25 info: do_dc_takeover: Taking over DC status for this partition cib[3266]: 2009/07/24_11:05:25 info: revision_check: Updating CIB revision to 2.0 cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: - <cib epoch="0"/> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + <cib epoch="1"> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + <configuration> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + <crm_config> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + <cluster_property_set id="cib-bootstrap-options"> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + <attributes> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="2.1.3-node: 552305612591183b1628baa5bc6e903e0f1e26a3"/> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + </attributes> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + </cluster_property_set> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + </crm_config> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + </configuration> cib[3266]: 2009/07/24_11:05:25 info: log_data_element: cib:diff: + </cib> cib[3310]: 2009/07/24_11:05:25 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3310]: 2009/07/24_11:05:25 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3310]: 2009/07/24_11:05:25 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3270]: 2009/07/24_11:05:25 info: join_make_offer: Making join offers based on membership 1 crmd[3270]: 2009/07/24_11:05:25 info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks tengine[3308]: 2009/07/24_11:05:25 info: G_main_add_SignalHandler: Added signal handler for signal 15 tengine[3308]: 2009/07/24_11:05:25 info: G_main_add_TriggerHandler: Added signal manual handler tengine[3308]: 2009/07/24_11:05:25 info: G_main_add_TriggerHandler: Added signal manual handler cib[3266]: 2009/07/24_11:05:25 info: cib_null_callback: Setting cib_diff_notify callbacks for tengine: on tengine[3308]: 2009/07/24_11:05:25 info: te_init: Registering TE UUID: 649f4c3c-a203-488e-bc62-91a34106a8cb tengine[3308]: 2009/07/24_11:05:25 info: set_graph_functions: Setting custom graph functions tengine[3308]: 2009/07/24_11:05:25 info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses tengine[3308]: 2009/07/24_11:05:25 info: te_init: Starting tengine tengine[3308]: 2009/07/24_11:05:25 info: te_connect_stonith: Attempting connection to fencing daemon... pengine[3309]: 2009/07/24_11:05:25 info: G_main_add_SignalHandler: Added signal handler for signal 15 pengine[3309]: 2009/07/24_11:05:25 info: pe_init: Starting pengine cib[3310]: 2009/07/24_11:05:25 info: write_cib_contents: Wrote version 0.1.1 of the CIB to disk (digest: e0890b65fda4792e35d6a3c8ba7b6da2) cib[3310]: 2009/07/24_11:05:25 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) cib[3310]: 2009/07/24_11:05:25 info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: /var/lib/heartbeat/crm/cib.xml.sig.last) crmd[3270]: 2009/07/24_11:05:26 info: update_dc: Set DC to haproxy1.b.cel.brightcove.com (2.0) crmd[3270]: 2009/07/24_11:05:26 info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] crmd[3270]: 2009/07/24_11:05:26 info: do_state_transition: All 1 cluster nodes responded to the join offer. crmd[3270]: 2009/07/24_11:05:26 info: update_attrd: Connecting to attrd... attrd[3269]: 2009/07/24_11:05:26 info: attrd_local_callback: Sending full refresh cib[3266]: 2009/07/24_11:05:26 info: sync_our_cib: Syncing CIB to all peers crmd[3270]: 2009/07/24_11:05:26 info: update_dc: Set DC to haproxy1.b.cel.brightcove.com (2.0) tengine[3308]: 2009/07/24_11:05:26 info: te_connect_stonith: Connected crmd[3270]: 2009/07/24_11:05:27 info: do_dc_join_ack: join-1: Updating node state to member for haproxy1.b.cel.brightcove.com crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: All 1 cluster nodes are eligible to run resources. tengine[3308]: 2009/07/24_11:05:27 info: update_abort_priority: Abort priority upgraded to 1000000 tengine[3308]: 2009/07/24_11:05:27 info: update_abort_priority: 'DC Takeover' abort superceeded pengine[3309]: 2009/07/24_11:05:27 info: determine_online_status: Node haproxy1.b.cel.brightcove.com is online crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] tengine[3308]: 2009/07/24_11:05:27 info: unpack_graph: Unpacked transition 0: 1 actions in 1 synapses tengine[3308]: 2009/07/24_11:05:27 info: send_rsc_command: Initiating action 2: probe_complete on haproxy1.b.cel.brightcove.com tengine[3308]: 2009/07/24_11:05:27 info: run_graph: Transition 0: (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0) tengine[3308]: 2009/07/24_11:05:27 info: notify_crmd: Transition 0 status: te_complete - <null> crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] tengine[3308]: 2009/07/24_11:05:27 info: extract_event: Aborting on transient_attributes changes for 7285647b-01fe-48b3-adc2-b12250b98c96 tengine[3308]: 2009/07/24_11:05:27 info: update_abort_priority: Abort priority upgraded to 1000000 crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ] crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: All 1 cluster nodes are eligible to run resources. pengine[3309]: 2009/07/24_11:05:27 info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-0.bz2 crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] tengine[3308]: 2009/07/24_11:05:27 info: unpack_graph: Unpacked transition 1: 0 actions in 0 synapses pengine[3309]: 2009/07/24_11:05:27 info: determine_online_status: Node haproxy1.b.cel.brightcove.com is online tengine[3308]: 2009/07/24_11:05:27 info: run_graph: Transition 1: (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0) tengine[3308]: 2009/07/24_11:05:27 info: notify_crmd: Transition 1 status: te_complete - <null> crmd[3270]: 2009/07/24_11:05:27 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] pengine[3309]: 2009/07/24_11:05:27 info: process_pe_message: Transition 1: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-1.bz2 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jul 24, 2009 at 10:37 PM, Andrew Beekhof <[email protected]> wrote: > On Fri, Jul 24, 2009 at 4:34 PM, Jiayin Mao<[email protected]> wrote: > > Do you mean "autojoin any"? > > right > > > I had it there, but heartbeat still complained > > the new node is not in their membership. > > logs? > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems > -- Max Mao (Mao Jia Yin) Abao Scrum Team, Engineering Department ----------------------------------------------------------- I am located at Beijing office, and usually work during 9:00PM and 6:00AM ET. _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
