Hi, I'm trying to setup a 3 nodes cluster on 3 physical machines. Two of them (name them S and T) are Debian squeeze freshly installed, the third is a CentOS 5.6 (node F)
All of them run pacemaker 1.0.11 for squeeze-testing : corosync 1.2.1-4 openais 1.1.2-2 pacemaker 1.0.11-1~bpo60+1 It seems that I'm able to bring up corosync for F and S or for F and T. But as soon as I start corosync on the 3rd node (T for instance), the corosync process dies either on the other (S) or on the current machine (T) So what, are S and T hating each other ? Did I miss a configuration parameter ? I attached the logs from machine sotalie : I started corosync, it seemed to see both other nodes, and dies with OpenAIS complains. ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Success (0) If you have any idea on what's happening, I'd be happy to here from you. Thanks, Jonathan -- IUEM - Service Informatique rue Dumont D'Urville Technopôle Brest-Iroise 29280 Plouzané France tel: +33 2 98 49 87 94
0xA8657ED2.asc
Description: application/pgp-keys
Jul 28 10:56:54 sotalie corosync[9875]: [MAIN ] Corosync Cluster Engine ('1.2.1'): started and ready to provide service.
Jul 28 10:56:54 sotalie corosync[9875]: [MAIN ] Corosync built-in features: nss
Jul 28 10:56:54 sotalie corosync[9875]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.
Jul 28 10:56:54 sotalie corosync[9875]: [TOTEM ] Initializing transport (UDP/IP).
Jul 28 10:56:54 sotalie corosync[9875]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Jul 28 10:56:54 sotalie corosync[9875]: [TOTEM ] The network interface [172.16.2.246] is now up.
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: process_ais_conf: Reading configure
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: config_find_init: Local handle: 4858364909567606786 for logging
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: config_find_next: Processing additional logging options...
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: get_config_opt: Found 'off' for option: debug
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: get_config_opt: Found 'no' for option: to_logfile
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: get_config_opt: Found 'local1' for option: syslog_facility
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: config_find_init: Local handle: 7101201744981393411 for service
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: config_find_next: Processing additional service options...
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_startup: CRM: Initialized
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] Logging: Initialized pcmk_startup
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_startup: Service: 9
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_startup: Local hostname: sotalie
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_update_nodeid: Local node id: 1979846828
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: update_member: Creating entry for node 1979846828 born on 0
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: update_member: 0xeecfb0 Node 1979846828 now known as sotalie (was: (null))
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: update_member: Node sotalie now has 1 quorum votes (was 0)
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: update_member: Node 1979846828/sotalie is now: member
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: spawn_child: Forked child 9886 for process stonithd
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: spawn_child: Forked child 9887 for process cib
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: spawn_child: Forked child 9888 for process lrmd
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: spawn_child: Forked child 9889 for process attrd
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: spawn_child: Forked child 9890 for process pengine
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: spawn_child: Forked child 9891 for process crmd
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.0.11
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: openais checkpoint service B.01.01
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: corosync extended virtual synchrony service
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: corosync configuration service
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: corosync cluster config database access v1.01
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: corosync profile loading service
Jul 28 10:56:54 sotalie corosync[9875]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1
Jul 28 10:56:54 sotalie corosync[9875]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
Jul 28 10:56:54 sotalie stonithd: [9886]: WARN: Core dumps could be lost if multiple dumps occur.
Jul 28 10:56:54 sotalie stonithd: [9886]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Jul 28 10:56:54 sotalie stonithd: [9886]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Jul 28 10:56:54 sotalie lrmd: [9888]: info: Signal sent to pid=9599, waiting for process to exit
Jul 28 10:56:54 sotalie lrmd: [9599]: info: lrmd is shutting down
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4208: memb=0, new=0, lost=0
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4208: memb=1, new=1, lost=0
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: NEW: sotalie 1979846828
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: MEMB: sotalie 1979846828
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: update_member: Node sotalie now has process list: 00000000000000000000000000013312 (78610)
Jul 28 10:56:54 sotalie corosync[9875]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jul 28 10:56:54 sotalie cib: [9887]: info: Invoked: /usr/lib/heartbeat/cib
Jul 28 10:56:54 sotalie cib: [9887]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)
Jul 28 10:56:54 sotalie attrd: [9889]: info: Invoked: /usr/lib/heartbeat/attrd
Jul 28 10:56:54 sotalie crmd: [9891]: info: Invoked: /usr/lib/heartbeat/crmd
Jul 28 10:56:54 sotalie pengine: [9890]: info: Invoked: /usr/lib/heartbeat/pengine
Jul 28 10:56:54 sotalie crmd: [9891]: info: main: CRM Hg Version: 6e010d6b0d49a6b929d17c0114e9d2d934dc8e04
Jul 28 10:56:54 sotalie attrd: [9889]: info: main: Starting up
Jul 28 10:56:54 sotalie crmd: [9891]: info: crmd_init: Starting crmd
Jul 28 10:56:54 sotalie pengine: [9890]: WARN: main: Terminating previous PE instance
Jul 28 10:56:54 sotalie attrd: [9889]: info: crm_cluster_connect: Connecting to OpenAIS
Jul 28 10:56:54 sotalie attrd: [9889]: info: init_ais_connection_once: Creating connection to our AIS plugin
Jul 28 10:56:54 sotalie pengine: [9601]: WARN: process_pe_message: Received quit message, terminating
Jul 28 10:56:54 sotalie corosync[9875]: [MAIN ] Completed service synchronization, ready to provide service.
Jul 28 10:56:54 sotalie attrd: [9889]: info: init_ais_connection_once: AIS connection established
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_ipc: Recorded connection 0xefa3f0 for attrd/9889
Jul 28 10:56:54 sotalie attrd: [9889]: info: get_ais_nodeid: Server details: id=1979846828 uname=sotalie cname=pcmk
Jul 28 10:56:54 sotalie attrd: [9889]: info: crm_new_peer: Node sotalie now has id: 1979846828
Jul 28 10:56:54 sotalie attrd: [9889]: info: crm_new_peer: Node 1979846828 is now known as sotalie
Jul 28 10:56:54 sotalie attrd: [9889]: info: main: Cluster connection active
Jul 28 10:56:54 sotalie attrd: [9889]: info: main: Accepting attribute updates
Jul 28 10:56:54 sotalie attrd: [9889]: info: main: Starting mainloop...
Jul 28 10:56:54 sotalie stonithd: [9886]: info: crm_cluster_connect: Connecting to OpenAIS
Jul 28 10:56:54 sotalie stonithd: [9886]: info: init_ais_connection_once: Creating connection to our AIS plugin
Jul 28 10:56:54 sotalie stonithd: [9886]: info: init_ais_connection_once: AIS connection established
Jul 28 10:56:54 sotalie cib: [9887]: info: startCib: CIB Initialization completed successfully
Jul 28 10:56:54 sotalie cib: [9887]: info: crm_cluster_connect: Connecting to OpenAIS
Jul 28 10:56:54 sotalie cib: [9887]: info: init_ais_connection_once: Creating connection to our AIS plugin
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_ipc: Recorded connection 0xefc050 for stonithd/9886
Jul 28 10:56:54 sotalie stonithd: [9886]: info: get_ais_nodeid: Server details: id=1979846828 uname=sotalie cname=pcmk
Jul 28 10:56:54 sotalie cib: [9887]: info: init_ais_connection_once: AIS connection established
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_ipc: Recorded connection 0xefcc40 for cib/9887
Jul 28 10:56:54 sotalie stonithd: [9886]: info: crm_new_peer: Node sotalie now has id: 1979846828
Jul 28 10:56:54 sotalie corosync[9875]: [pcmk ] info: pcmk_ipc: Sending membership update 4208 to cib
Jul 28 10:56:54 sotalie stonithd: [9886]: info: crm_new_peer: Node 1979846828 is now known as sotalie
Jul 28 10:56:54 sotalie cib: [9887]: info: get_ais_nodeid: Server details: id=1979846828 uname=sotalie cname=pcmk
Jul 28 10:56:54 sotalie cib: [9887]: info: crm_new_peer: Node sotalie now has id: 1979846828
Jul 28 10:56:54 sotalie cib: [9887]: info: crm_new_peer: Node 1979846828 is now known as sotalie
Jul 28 10:56:54 sotalie stonithd: [9886]: notice: /usr/lib/heartbeat/stonithd start up successfully.
Jul 28 10:56:54 sotalie cib: [9887]: info: cib_init: Starting cib mainloop
Jul 28 10:56:54 sotalie cib: [9887]: info: ais_dispatch: Membership 4208: quorum still lost
Jul 28 10:56:54 sotalie cib: [9887]: info: crm_update_peer: Node sotalie: id=1979846828 state=member (new) addr=r(0) ip(172.16.2.246) (new) votes=1 (new) born=0 seen=4208 proc=00000000000000000000000000013312 (new)
Jul 28 10:56:54 sotalie cib: [9895]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-8.raw
Jul 28 10:56:54 sotalie cib: [9895]: info: write_cib_contents: Wrote version 0.40.0 of the CIB to disk (digest: eebcbbc34fe0b71d67d0df995f215526)
Jul 28 10:56:54 sotalie cib: [9895]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.fQptLS (digest: /var/lib/heartbeat/crm/cib.srs5pB)
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4216: memb=1, new=0, lost=0
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: memb: sotalie 1979846828
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4216: memb=3, new=2, lost=0
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Creating entry for node 318902444 born on 4216
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Node 318902444/unknown is now: member
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 318902444
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Creating entry for node 2013401260 born on 4216
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Node 2013401260/unknown is now: member
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2013401260
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 318902444
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: MEMB: sotalie 1979846828
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2013401260
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: send_member_notification: Sending membership update 4216 to 1 children
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: 0xeecfb0 Node 1979846828 (sotalie) born on: 4216
Jul 28 10:56:55 sotalie cib: [9887]: info: ais_dispatch: Membership 4216: quorum still lost
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_new_peer: Node <null> now has id: 318902444
Jul 28 10:56:55 sotalie corosync[9875]: [TOTEM ] A processor joined or left the membership and a new membership was formed.
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_update_peer: Node (null): id=318902444 state=member (new) addr=r(0) ip(172.16.2.19) votes=0 born=0 seen=4216 proc=00000000000000000000000000000000
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_new_peer: Node <null> now has id: 2013401260
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_update_peer: Node (null): id=2013401260 state=member (new) addr=r(0) ip(172.16.2.248) votes=0 born=0 seen=4216 proc=00000000000000000000000000000000
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: 0xefd690 Node 2013401260 ((null)) born on: 4176
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: 0xefd690 Node 2013401260 now known as tursiop (was: (null))
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Node tursiop now has process list: 00000000000000000000000000013312 (78610)
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Node tursiop now has 1 quorum votes (was 0)
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: send_member_notification: Sending membership update 4216 to 1 children
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
Jul 28 10:56:55 sotalie cib: [9887]: notice: ais_dispatch: Membership 4216: quorum acquired
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_get_peer: Node 2013401260 is now known as tursiop
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_update_peer: Node tursiop: id=2013401260 state=member addr=r(0) ip(172.16.2.248) votes=1 (new) born=4176 seen=4216 proc=00000000000000000000000000013312 (new)
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: 0xefad60 Node 318902444 ((null)) born on: 4212
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: 0xefad60 Node 318902444 now known as feiri-test2 (was: (null))
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Node feiri-test2 now has process list: 00000000000000000000000000013312 (78610)
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_member: Node feiri-test2 now has 1 quorum votes (was 0)
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: update_expected_votes: Expected quorum votes 2 -> 3
Jul 28 10:56:55 sotalie corosync[9875]: [pcmk ] info: send_member_notification: Sending membership update 4216 to 1 children
Jul 28 10:56:55 sotalie cib: [9887]: info: ais_dispatch: Membership 4216: quorum retained
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_get_peer: Node 318902444 is now known as feiri-test2
Jul 28 10:56:55 sotalie cib: [9887]: info: crm_update_peer: Node feiri-test2: id=318902444 state=member addr=r(0) ip(172.16.2.19) votes=1 (new) born=4212 seen=4216 proc=00000000000000000000000000013312 (new)
Jul 28 10:56:55 sotalie cib: [9887]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Success (0)
Jul 28 10:56:55 sotalie cib: [9887]: ERROR: ais_dispatch: AIS connection failed
Jul 28 10:56:55 sotalie cib: [9887]: ERROR: cib_ais_destroy: AIS connection terminated
Jul 28 10:56:55 sotalie attrd: [9889]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Success (0)
Jul 28 10:56:55 sotalie attrd: [9889]: ERROR: ais_dispatch: AIS connection failed
Jul 28 10:56:55 sotalie attrd: [9889]: CRIT: attrd_ais_destroy: Lost connection to OpenAIS service!
Jul 28 10:56:55 sotalie attrd: [9889]: info: main: Exiting...
Jul 28 10:56:55 sotalie stonithd: [9886]: ERROR: ais_dispatch: Receiving message body failed: (2) Library error: Success (0)
Jul 28 10:56:55 sotalie stonithd: [9886]: ERROR: ais_dispatch: AIS connection failed
Jul 28 10:56:55 sotalie stonithd: [9886]: ERROR: AIS connection terminated
Jul 28 10:56:55 sotalie lrmd: [9888]: info: enabling coredumps
Jul 28 10:56:55 sotalie lrmd: [9888]: WARN: Core dumps could be lost if multiple dumps occur.
Jul 28 10:56:55 sotalie lrmd: [9888]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability
Jul 28 10:56:55 sotalie lrmd: [9888]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability
Jul 28 10:56:55 sotalie lrmd: [9888]: info: Started.
Jul 28 10:56:55 sotalie crmd: [9891]: info: do_cib_control: Could not connect to the CIB service: connection failed
Jul 28 10:56:55 sotalie crmd: [9891]: WARN: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry
Jul 28 10:56:55 sotalie crmd: [9891]: info: crmd_init: Starting crmd's mainloop
Jul 28 10:56:56 sotalie pengine: [9890]: info: main: Starting pengine
Jul 28 10:56:57 sotalie crmd: [9891]: info: crm_timer_popped: Wait Timer (I_NULL) just popped!
Jul 28 10:56:58 sotalie crmd: [9891]: info: do_cib_control: Could not connect to the CIB service: connection failed
Jul 28 10:56:58 sotalie crmd: [9891]: WARN: do_cib_control: Couldn't complete CIB registration 2 times... pause and retry
Jul 28 10:57:00 sotalie crmd: [9891]: info: crm_timer_popped: Wait Timer (I_NULL) just popped!
Jul 28 10:57:01 sotalie crmd: [9891]: info: do_cib_control: Could not connect to the CIB service: connection failed
Jul 28 10:57:01 sotalie crmd: [9891]: WARN: do_cib_control: Couldn't complete CIB registration 3 times... pause and retry
Jul 28 10:57:03 sotalie crmd: [9891]: info: crm_timer_popped: Wait Timer (I_NULL) just popped!
Jul 28 10:57:04 sotalie crmd: [9891]: info: do_cib_control: Could not connect to the CIB service: connection failed
Jul 28 10:57:04 sotalie crmd: [9891]: WARN: do_cib_control: Couldn't complete CIB registration 4 times... pause and retry
Jul 28 10:57:06 sotalie crmd: [9891]: info: crm_timer_popped: Wait Timer (I_NULL) just popped!
Jul 28 10:57:07 sotalie crmd: [9891]: info: do_cib_control: Could not connect to the CIB service: connection failed
Jul 28 10:57:07 sotalie crmd: [9891]: WARN: do_cib_control: Couldn't complete CIB registration 5 times... pause and retry
Jul 28 10:57:09 sotalie crmd: [9891]: info: crm_timer_popped: Wait Timer (I_NULL) just popped!
Jul 28 10:57:10 sotalie crmd: [9891]: info: do_cib_control: Could not connect to the CIB service: connection failed
Jul 28 10:57:10 sotalie crmd: [9891]: WARN: do_cib_control: Couldn't complete CIB registration 6 times... pause and retry
Jul 28 10:57:12 sotalie crmd: [9891]: info: crm_timer_popped: Wait Timer (I_NULL) just popped!
Jul 28 10:57:13 sotalie crmd: [9891]: info: do_cib_control: Could not connect to the CIB service: connection failed
Jul 28 10:57:13 sotalie crmd: [9891]: WARN: do_cib_control: Couldn't complete CIB registration 7 times... pause and retry
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
