Hi. After running a cluster node in a production cluster since July, I got the folllowing error:
<err> #48: Unable to obtain cluster lock: Invalid argument Which resulted in a reboot: --clip-- Dec 27 02:50:31 pcn1 clurgmgrd[6217]: <err> #48: Unable to obtain cluster lock: Invalid argument Dec 27 02:50:31 pcn1 clurgmgrd[6217]: <notice> Stopping service service:p01 Dec 27 02:50:34 pcn1 in.rdiscd[30325]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use Dec 27 02:50:34 pcn1 in.rdiscd[30325]: Failed joining addresses Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer 'ia_addr' insert (-1) Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer 'ia_addr' insert (-1) Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer '' insert (-1) Dec 27 02:50:38 pcn1 snmpd[15929]: error on subcontainer '' insert (-1) Dec 27 02:50:45 pcn1 clurgmgrd[6217]: <notice> Service service:p01 is recovering Dec 27 02:50:45 pcn1 clurgmgrd[6217]: <notice> Recovering failed service service:p01 Dec 27 02:50:45 pcn1 kernel: dlm: add_to_waiters error 1 Dec 27 02:50:45 pcn1 kernel: dlm: remove_from_waiters error Dec 27 02:50:45 pcn1 kernel: dlm: rgmanager: receive_unlock_reply not on waiters Dec 27 02:50:45 pcn1 clurgmgrd[6216]: <crit> Watchdog: Daemon died, rebooting... Dec 27 02:50:45 pcn1 kernel: md: stopping all md devices. Dec 27 02:55:23 pcn1 syslogd 1.4.1: restart. --clip-- Other members of the cluster noticed the missing member, fenced it, failed services over, and back (when the missing node had rejoined): --clip-- Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] The token was lost in the OPERATIONAL state. Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Dec 27 02:50:56 pcn2 openais[4588]: [TOTEM] entering GATHER state from 2. Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering GATHER state from 11. Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Saving state aru 6a4 high seq received 6a4 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering COMMIT state. Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering RECOVERY state. Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [0] member 10.3.0.10: Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [1] member 10.3.0.12: Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [2] member 10.3.0.13: Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [3] member 10.3.0.14: Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 0 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [4] member 10.3.0.15: Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 1 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] position [5] member 10.3.0.16: Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] previous ring seq 324 rep 10.3.0.10 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] aru 6a4 high delivered 6a4 received flag 1 Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Did not need to originate any messages in recovery. Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] Storing new sequence id for ring 14c Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] New Configuration: Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.16) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Left: Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.11) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Joined: Dec 27 02:51:01 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] New Configuration: Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.16) Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Left: Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] Members Joined: Dec 27 02:51:01 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:51:01 pcn2 openais[4588]: [TOTEM] entering OPERATIONAL state. Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.10 Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.12 Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.13 Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.14 Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.15 Dec 27 02:51:01 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.16 Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 3 Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 4 Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 5 Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 6 Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 100 Dec 27 02:51:01 pcn2 openais[4588]: [CPG ] got joinlist message from node 2 Dec 27 02:51:01 pcn2 kernel: dlm: closing connection to node 1 Dec 27 02:51:01 pcn2 fenced[4614]: pcn1-hb not a cluster member after 0 sec post_fail_delay Dec 27 02:51:01 pcn2 fenced[4614]: fencing node "pcn1-hb" Dec 27 02:52:13 pcn2 fenced[4614]: fence "pcn1-hb" success Dec 27 02:52:18 pcn2 ccsd[4541]: Attempt to close an unopened CCS descriptor (799075500). Dec 27 02:52:18 pcn2 ccsd[4541]: Error while processing disconnect: Invalid request descriptor Dec 27 02:52:20 pcn2 clurgmgrd[6262]: <notice> Taking over service service:p01 from down member pcn1-hb Dec 27 02:52:20 pcn2 clurgmgrd[6262]: <notice> Taking over service service:i01 from down member pcn1-hb Dec 27 02:52:20 pcn2 kernel: kjournald starting. Commit interval 5 seconds Dec 27 02:52:20 pcn2 kernel: EXT3 FS on dm-65, internal journal Dec 27 02:52:20 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Dec 27 02:52:21 pcn2 clurgmgrd[6262]: <notice> Taking over service service:i13 from down member pcn1-hb Dec 27 02:52:21 pcn2 in.rdiscd[2158]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use Dec 27 02:52:21 pcn2 in.rdiscd[2158]: Failed joining addresses Dec 27 02:52:22 pcn2 kernel: kjournald starting. Commit interval 5 seconds Dec 27 02:52:22 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended Dec 27 02:52:22 pcn2 kernel: EXT3 FS on dm-14, internal journal Dec 27 02:52:22 pcn2 kernel: EXT3-fs: recovery complete. Dec 27 02:52:22 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Dec 27 02:52:24 pcn2 clurgmgrd[6262]: <notice> Service service:p01 started Dec 27 02:52:25 pcn2 last message repeated 2 times Dec 27 02:52:27 pcn2 kernel: kjournald starting. Commit interval 5 seconds Dec 27 02:52:27 pcn2 kernel: EXT3 FS on dm-2, internal journal Dec 27 02:52:27 pcn2 kernel: EXT3-fs: dm-2: 3 orphan inodes deleted Dec 27 02:52:27 pcn2 kernel: EXT3-fs: recovery complete. Dec 27 02:52:27 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Dec 27 02:52:29 pcn2 kernel: kjournald starting. Commit interval 5 seconds Dec 27 02:52:29 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended Dec 27 02:52:29 pcn2 kernel: EXT3 FS on dm-38, internal journal Dec 27 02:52:29 pcn2 kernel: EXT3-fs: recovery complete. Dec 27 02:52:29 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Dec 27 02:52:30 pcn2 in.rdiscd[3313]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use Dec 27 02:52:30 pcn2 in.rdiscd[3313]: Failed joining addresses Dec 27 02:52:32 pcn2 clurgmgrd[6262]: <notice> Service service:i13 started Dec 27 02:52:35 pcn2 kernel: kjournald starting. Commit interval 5 seconds Dec 27 02:52:35 pcn2 kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended Dec 27 02:52:35 pcn2 kernel: EXT3 FS on dm-26, internal journal Dec 27 02:52:35 pcn2 kernel: EXT3-fs: recovery complete. Dec 27 02:52:35 pcn2 kernel: EXT3-fs: mounted filesystem with ordered data mode. Dec 27 02:52:37 pcn2 in.rdiscd[3833]: setsockopt (IP_ADD_MEMBERSHIP): Address already in use Dec 27 02:52:37 pcn2 in.rdiscd[3833]: Failed joining addresses Dec 27 02:52:38 pcn2 clurgmgrd[6262]: <notice> Service service:i01 started Dec 27 02:53:25 pcn2 last message repeated 2 times Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering GATHER state from 11. Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Saving state aru c8 high seq received c8 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering COMMIT state. Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering RECOVERY state. Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [0] member 10.3.0.10: Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [1] member 10.3.0.11: Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 288 rep 10.3.0.11 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru 9 high delivered 9 received flag 0 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [2] member 10.3.0.12: Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [3] member 10.3.0.13: Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [4] member 10.3.0.14: Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [5] member 10.3.0.15: Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 1 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] position [6] member 10.3.0.16: Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] aru c8 high delivered c8 received flag 1 Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Did not need to originate any messages in recovery. Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] Storing new sequence id for ring 150 Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] New Configuration: Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Left: Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Joined: Dec 27 02:55:26 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] New Configuration: Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.10) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.11) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.12) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.13) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.14) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.15) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.16) Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Left: Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] Members Joined: Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] r(0) ip(10.3.0.11) Dec 27 02:55:26 pcn2 openais[4588]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:55:26 pcn2 openais[4588]: [TOTEM] entering OPERATIONAL state. Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.10 Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.11 Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.12 Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.13 Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.14 Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.15 Dec 27 02:55:26 pcn2 openais[4588]: [CLM ] got nodejoin message 10.3.0.16 Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 100 Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 2 Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 3 Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 4 Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 5 Dec 27 02:55:26 pcn2 openais[4588]: [CPG ] got joinlist message from node 6 Dec 27 02:55:35 pcn2 kernel: dlm: connecting to 1 --clip-- --clip-- Dec 27 02:55:24 pcn1 ccsd[4132]: Starting ccsd 2.0.69: Dec 27 02:55:24 pcn1 ccsd[4132]: Built: Jun 27 2007 15:21:32 Dec 27 02:55:24 pcn1 ccsd[4132]: Copyright (C) Red Hat, Inc. 2004 All rights reserved. Dec 27 02:55:24 pcn1 ccsd[4132]: cluster.conf (cluster name = mappi-primary, version = 109) found. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] AIS Executive Service RELEASE 'subrev 1324 version 0.80.2' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contribu tors. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Copyright (C) 2006 Red Hat, Inc. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] AIS Executive Service: started and ready to provide service. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Using default multicast address of 239.192.46.199 Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cpg loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais cluster closed process g roup service v1.01' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cfg loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais configuration service' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_msg loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais message service B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_lck loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais distributed locking serv ice B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_evt loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais event service B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_ckpt loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais checkpoint service B.01. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_amf loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais availability management framework B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_clm loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais cluster membership servi ce B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_evs loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais extended virtual synchro ny service' Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] openais component openais_cman loaded. Dec 27 02:55:26 pcn1 openais[4143]: [MAIN ] Registering service handler 'openais CMAN membership service 2.01' Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] token hold (386 ms) retransmits before loss (20 retrans) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] window size per rotation (50 messages) maximum messages per r otation (17 messages) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] send threads (0 threads) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP token expired timeout (495 ms) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP token problem counter (2000 ms) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP threshold (10 problem count) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] RRP mode set to none. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] heartbeat_failures_allowed (0) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] max_network_delay (50 ms) Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allow ed > 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Receive multicast socket recv buffer size (262142 bytes). Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes). Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] The network interface [10.3.0.11] is now up. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Created or loaded sequence id 284.10.3.0.11 for this ring. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering GATHER state from 15. Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais extended virtual synchr ony service' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais cluster membership serv ice B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais availability management framework B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais checkpoint service B.01 .01' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais event service B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais distributed locking ser vice B.01.01' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais message service B.01.01 ' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais configuration service' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' Dec 27 02:55:26 pcn1 openais[4143]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01' Dec 27 02:55:26 pcn1 openais[4143]: [CMAN ] CMAN 2.0.69 (built Jun 27 2007 15:21:36) started Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] Not using a virtual synchrony filter. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Creating commit token because I am the rep. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Saving state aru 0 high seq received 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering COMMIT state. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering RECOVERY state. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [0] member 10.3.0.11: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 284 rep 10.3.0.11 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru 0 high delivered 0 received flag 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Did not need to originate any messages in recovery. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Storing new sequence id for ring 120 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Sending initial ORF token Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined: Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11) Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering OPERATIONAL state. Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.11 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering GATHER state from 11. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Saving state aru 9 high seq received 9 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering COMMIT state. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering RECOVERY state. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [0] member 10.3.0.10: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [1] member 10.3.0.11: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 288 rep 10.3.0.11 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru 9 high delivered 9 received flag 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [2] member 10.3.0.12: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [3] member 10.3.0.13: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [4] member 10.3.0.14: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 0 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [5] member 10.3.0.15: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 1 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] position [6] member 10.3.0.16: Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] previous ring seq 332 rep 10.3.0.10 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] aru c8 high delivered c8 received flag 1 Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Did not need to originate any messages in recovery. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] Storing new sequence id for ring 150 Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined: Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] CLM CONFIGURATION CHANGE Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] New Configuration: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.10) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.11) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.12) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.13) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.14) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.15) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.16) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Left: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] Members Joined: Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.10) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.12) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.13) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.14) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.15) Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] r(0) ip(10.3.0.16) Dec 27 02:55:26 pcn1 openais[4143]: [SYNC ] This node is within the primary component and will provide se rvice. Dec 27 02:55:26 pcn1 openais[4143]: [TOTEM] entering OPERATIONAL state. Dec 27 02:55:26 pcn1 openais[4143]: [CMAN ] quorum regained, resuming activity Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.10 Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.11 Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.12 Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.13 Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.14 Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.15 Dec 27 02:55:26 pcn1 openais[4143]: [CLM ] got nodejoin message 10.3.0.16 Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 100 Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 2 Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 3 Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 4 Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 5 Dec 27 02:55:26 pcn1 openais[4143]: [CPG ] got joinlist message from node 6 Dec 27 02:55:26 pcn1 ccsd[4132]: Initial status:: Quorate Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 100 Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 2 Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 3 Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 5 Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 6 Dec 27 02:55:35 pcn1 kernel: dlm: got connection from 4 Dec 27 02:55:35 pcn1 clvmd: Cluster LVM daemon started - connected to CMAN Dec 27 03:01:04 pcn1 clurgmgrd[5515]: <notice> Starting stopped service service:i03 Dec 27 03:01:04 pcn1 clurgmgrd[5515]: <notice> Starting stopped service service:i15 [etc] --clip-- Now I tried googling around for the mysterious error message #48, and couldn't find any info. What might've been up? --Janne -- Janne Peltonen <[EMAIL PROTECTED]> -- Linux-cluster mailing list [email protected] https://www.redhat.com/mailman/listinfo/linux-cluster
