Hi Had another node die
Everything is looking good, I am guessing corrosync tried to talk to the other node and it failed, I believe Nov 1 00:08:48 demorp2 ntpd[2461]: peers refreshed Nov 1 00:08:51 demorp2 corosync[2039]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Nov 1 00:08:51 demorp2 corosync[2039]: [CPG ] chosen downlist: sender r(0) ip(10.172.218.52) ; members(old:1 left:0) Nov 1 00:08:51 demorp2 corosync[2039]: [MAIN ] Completed service synchronization, ready to provide service. Nov 1 00:09:05 demorp2 corosync[2039]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Nov 1 00:09:05 demorp2 corosync[2039]: [CMAN ] quorum regained, resuming activity Nov 1 00:09:05 demorp2 corosync[2039]: [QUORUM] This node is within the primary component and will provide service. Nov 1 00:09:05 demorp2 corosync[2039]: [QUORUM] Members[2]: 1 2 Nov 1 00:09:05 demorp2 corosync[2039]: [QUORUM] Members[2]: 1 2 Nov 1 00:09:05 demorp2 crmd[2725]: notice: cman_event_callback: Membership 320: quorum acquired Nov 1 00:09:05 demorp2 crmd[2725]: notice: crm_update_peer_state: cman_event_callback: Node demorp1[1] - state is now member (was lost) Nov 1 00:09:05 demorp2 corosync[2039]: [CPG ] chosen downlist: sender r(0) ip(10.172.218.52) ; members(old:1 left:0) Nov 1 00:09:05 demorp2 corosync[2039]: [MAIN ] Completed service synchronization, ready to provide service. Nov 1 00:09:05 demorp2 crmd[2725]: notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=peer_update_callback ] Nov 1 00:09:06 demorp2 corosync[2039]: cman killed by node 1 because we were killed by cman_tool or other application Nov 1 00:09:06 demorp2 attrd[2723]: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Nov 1 00:09:06 demorp2 attrd[2723]: crit: attrd_cs_destroy: Lost connection to Corosync service! Nov 1 00:09:06 demorp2 attrd[2723]: notice: main: Exiting... Nov 1 00:09:06 demorp2 attrd[2723]: notice: main: Disconnecting client 0xdc3020, pid=2725... Nov 1 00:09:06 demorp2 pacemakerd[2712]: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Nov 1 00:09:06 demorp2 pacemakerd[2712]: error: mcp_cpg_destroy: Connection destroyed Nov 1 00:09:06 demorp2 stonith-ng[2721]: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Nov 1 00:09:06 demorp2 crmd[2725]: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Nov 1 00:09:06 demorp2 crmd[2725]: error: crmd_cs_destroy: connection terminated Nov 1 00:09:06 demorp2 gfs_controld[2173]: cluster is down, exiting Nov 1 00:09:06 demorp2 gfs_controld[2173]: daemon cpg_dispatch error 2 Nov 1 00:09:06 demorp2 attrd[2723]: error: attrd_cib_connection_destroy: Connection to the CIB terminated... Nov 1 00:09:06 demorp2 fenced[2098]: cluster is down, exiting Nov 1 00:09:06 demorp2 fenced[2098]: daemon cpg_dispatch error 2 Nov 1 00:09:06 demorp2 dlm_controld[2124]: cluster is down, exiting Nov 1 00:09:06 demorp2 dlm_controld[2124]: daemon cpg_dispatch error 2 Nov 1 00:09:06 demorp2 stonith-ng[2721]: error: stonith_peer_cs_destroy: Corosync connection terminated Nov 1 00:09:06 demorp2 cib[2720]: warning: qb_ipcs_event_sendv: new_event_notification (2720-2721-11): Broken pipe (32) Nov 1 00:09:06 demorp2 cib[2720]: warning: cib_notify_send_one: Notification of client crmd/4c1076bf-8a95-4f77-b866-e1bbf5e2ceda failed Nov 1 00:09:06 demorp2 cib[2720]: error: pcmk_cpg_dispatch: Connection to the CPG API failed: Library error (2) Nov 1 00:09:06 demorp2 cib[2720]: error: cib_cs_destroy: Corosync connection lost! Exiting. Nov 1 00:09:06 demorp2 crmd[2725]: notice: crmd_exit: Forcing immediate exit: Link has been severed (67) Nov 1 00:09:06 demorp2 lrmd[2722]: warning: qb_ipcs_event_sendv: new_event_notification (2722-2725-6): Bad file descriptor (9) Nov 1 00:09:06 demorp2 lrmd[2722]: warning: send_client_notify: Notification of client crmd/3598d3e2-600a-4f15-aae2-e087437d6213 failed Nov 1 00:09:06 demorp2 lrmd[2722]: warning: send_client_notify: Notification of client crmd/3598d3e2-600a-4f15-aae2-e087437d6213 failed Nov 1 00:09:08 demorp2 kernel: dlm: closing connection to node 1 The other node It looks to me, like VMWare took too long to give this vm a time slice and corosync responded by killing one node ov 1 00:08:50 demorp1 lrmd[2433]: warning: child_timeout_callback: ybrpstat_monitor_5000 process (PID 32026) timed out Nov 1 00:08:50 demorp1 lrmd[2433]: warning: operation_finished: ybrpstat_monitor_5000:32026 - timed out after 20000ms Nov 1 00:08:51 demorp1 crmd[2436]: error: process_lrm_event: LRM operation ybrpstat_monitor_5000 (17) Timed Out (timeout=20000ms) Nov 1 00:08:52 demorp1 crmd[2436]: notice: process_lrm_event: demorp1-ybrpstat_monitor_5000:17 [ Service running for 18 hours 8 minutes 30 seconds.\n ] Nov 1 00:08:53 demorp1 lrmd[2433]: warning: child_timeout_callback: ybrpip_monitor_5000 process (PID 32033) timed out Nov 1 00:08:53 demorp1 lrmd[2433]: warning: operation_finished: ybrpip_monitor_5000:32033 - timed out after 20000ms Nov 1 00:08:53 demorp1 crmd[2436]: error: process_lrm_event: LRM operation ybrpip_monitor_5000 (22) Timed Out (timeout=20000ms) Nov 1 00:09:05 demorp1 corosync[1748]: [MAIN ] Corosync main process was not scheduled for 16241.7002 ms (threshold is 8000.0000 ms). Consider token timeout increase. Nov 1 00:09:05 demorp1 corosync[1748]: [TOTEM ] A processor failed, forming new configuration. Nov 1 00:09:05 demorp1 corosync[1748]: [TOTEM ] Process pause detected for 15555 ms, flushing membership messages. Nov 1 00:09:05 demorp1 corosync[1748]: [MAIN ] Corosync main process was not scheduled for 15555.0029 ms (threshold is 8000.0000 ms). Consider token timeout increase. Nov 1 00:09:05 demorp1 corosync[1748]: [CMAN ] quorum lost, blocking activity Nov 1 00:09:05 demorp1 corosync[1748]: [QUORUM] This node is within the non-primary component and will NOT provide any services. Nov 1 00:09:05 demorp1 corosync[1748]: [QUORUM] Members[1]: 1 Nov 1 00:09:05 demorp1 corosync[1748]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Nov 1 00:09:05 demorp1 corosync[1748]: [CMAN ] quorum regained, resuming activity Nov 1 00:09:05 demorp1 corosync[1748]: [QUORUM] This node is within the primary component and will provide service. Nov 1 00:09:05 demorp1 corosync[1748]: [QUORUM] Members[2]: 1 2 Nov 1 00:09:05 demorp1 corosync[1748]: [QUORUM] Members[2]: 1 2 Nov 1 00:09:05 demorp1 corosync[1748]: [CPG ] chosen downlist: sender r(0) ip(10.172.218.51) ; members(old:2 left:1) Nov 1 00:09:05 demorp1 corosync[1748]: [MAIN ] Completed service synchronization, ready to provide service. Nov 1 00:09:05 demorp1 crmd[2436]: notice: process_lrm_event: LRM operation ybrpip_monitor_5000 (call=22, rc=0, cib-update=17, confirmed=false) ok Nov 1 00:09:05 demorp1 crmd[2436]: notice: peer_update_callback: Our peer on the DC is dead Nov 1 00:09:05 demorp1 crmd[2436]: notice: cman_event_callback: Membership 320: quorum lost Nov 1 00:09:05 demorp1 crmd[2436]: notice: cman_event_callback: Membership 320: quorum acquired Nov 1 00:09:05 demorp1 crmd[2436]: notice: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callba ck ] Nov 1 00:09:05 demorp1 crmd[2436]: notice: process_lrm_event: LRM operation ybrpstat_monitor_5000 (call=17, rc=0, cib-update=18, confirmed=false) ok Nov 1 00:09:06 demorp1 crmd[2436]: warning: do_log: FSA: Input I_JOIN_OFFER from route_message() received in state S_ELECTION Nov 1 00:09:06 demorp1 crmd[2436]: notice: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ] Nov 1 00:09:06 demorp1 fenced[1822]: telling cman to remove nodeid 2 from cluster Nov 1 00:09:06 demorp1 fenced[1822]: receive_start 2:3 add node with started_count 1 _______________________________________________ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
