On Mon, 2010-03-08 at 09:58 -0500, Joseph Salvaggio wrote: > We have a 2 node Red Hat cluster. Apache service and samba service, > for which we keep the GFS2 SAN partition always mounted on both nodes. > For the last few days been seeing "Retransmit List" a lot (over and > over, every 1-2 minutes or so) in our "messages" log > > What problem(s) is this possibly pointing to? > > Thank you. > > > from /var/log/messages > > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] The token was lost in > the OPERATIONAL state. > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Receive multicast > socket recv buffer size (320000 bytes). > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Transmit multicast > socket send buffer size (262142 bytes). > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering GATHER state > from 2. > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Creating commit token > because I am the rep. > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Saving state aru c136 > high seq received c136 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Storing new sequence > id for ring 48b80 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering COMMIT > state. > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY > state. > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] position [0] member > 192.168.100.7: > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] previous ring seq > 297852 rep 192.168.100.7 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] aru c136 high > delivered c136 received flag 1 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] position [1] member > 192.168.100.10: > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] previous ring seq > 297852 rep 192.168.100.7 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] aru c136 high > delivered c136 received flag 1 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Did not need to > originate any messages in recovery. > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF > token > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION > CHANGE > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] New Configuration: > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.7) > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.10) > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Left: > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Joined: > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION > CHANGE > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] New Configuration: > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.7) > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.10) > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Left: > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] Members Joined: > Mar 8 08:03:42 kaf-gimel openais[3973]: [SYNC ] This node is within > the primary component and will provide service. > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL > state. > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message > 192.168.100.7 > Mar 8 08:03:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message > 192.168.100.10 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 11 > 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 > 23 24 25 26 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20 > 21 18 19 1a 1b 1c 1d 1e 1f 22 23 24 25 26 > Mar 8 08:03:42 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry > 100 MSG_PLOCK > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1b > 1c 1d 1e 1f 20 21 22 23 24 25 26 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1f > 20 21 22 23 24 25 26 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20 > 21 22 23 24 25 26 > Mar 8 08:03:42 kaf-gimel last message repeated 4 times > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 > 23 24 25 26 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23 > 24 25 26 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 24 > 25 26 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 > 26 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 28 > 29 > Mar 8 08:03:42 kaf-gimel last message repeated 34 times > Mar 8 08:03:42 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry > 200 MSG_PLOCK > Mar 8 08:03:42 kaf-gimel openais[3973]: [CPG ] got joinlist message > from node 1 > Mar 8 08:03:42 kaf-gimel openais[3973]: [CPG ] got joinlist message > from node 2 > Mar 8 08:03:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 32 > Mar 8 08:40:46 kaf-gimel kernel: <=64 ID=6 ACK URGP=0 > Mar 8 08:40:46 kaf-gimel kernel: <710.0.100.21 LEN=52 TOS=0x00 > PREC=0x00 TTL=64 ID=41685 DF PROTO=TCP SPT=25 DPT=58398 WINDOW=64 > RES=0x00 ACK URGP=0 > Mar 8 08:55:55 kaf-gimel kernel: BANDWIDTH_IN:IN=eth0 OUT= > MAC=00:1b:21:46:fe:06:00:30:48:33:ad:b4:08:00 SRC=10.0.100.21 > DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 IT55 > WIS=0x00BANDWIDTH_IN:IN=eth0 OUT= > MAC=00:1b:21:46:fe:06:00:30:48:33:ad:b4:08:00 SRC=10.0.100.21 > DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=40663 DF PROTO=TCP > SPT=25 DPT=39615 WINDOW=46336 RES=0x00 ACK URGP=0 > Mar 8 08:55:55 kaf-gimel kernel: > WIDTH_IN:IN=eth0OUT=b:48:33:ad:b4:08:00 SRC=10.0.100.21 > DST=10.0.100.23 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=41095 DF PROTO=TCP > SPT=25 DPT=39615 WINDOW=47784 RES=0x00 ACK URGP=0 > Mar 8 09:00:48 kaf-gimel syslogd 1.4.1: restart. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] The token was lost in > the OPERATIONAL state. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Receive multicast > socket recv buffer size (320000 bytes). > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Transmit multicast > socket send buffer size (262142 bytes). > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering GATHER state > from 2. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Creating commit token > because I am the rep. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Saving state aru > 142c1 high seq received 142c1 > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Storing new sequence > id for ring 48b84 > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering COMMIT > state. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY > state. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] position [0] member > 192.168.100.7: > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] previous ring seq > 297856 rep 192.168.100.7 > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] aru 142c1 high > delivered 142c1 received flag 1 > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] position [1] member > 192.168.100.10: > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] previous ring seq > 297856 rep 192.168.100.7 > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] aru 142c1 high > delivered 142c1 received flag 1 > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Did not need to > originate any messages in recovery. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF > token > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION > CHANGE > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] New Configuration: > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.7) > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.10) > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Left: > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Joined: > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION > CHANGE > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] New Configuration: > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.7) > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.10) > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Left: > Mar 8 09:03:12 kaf-gimel openais[3973]: [CLM ] Members Joined: > Mar 8 09:03:12 kaf-gimel openais[3973]: [SYNC ] This node is within > the primary component and will provide service. > Mar 8 09:03:12 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL > state. > Mar 8 09:03:13 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry > 100 MSG_PLOCK > Mar 8 09:03:13 kaf-gimel openais[3973]: [CLM ] got nodejoin message > 192.168.100.7 > Mar 8 09:03:13 kaf-gimel openais[3973]: [CLM ] got nodejoin message > 192.168.100.10 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 15 > 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 26 > 27 28 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 19 > 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1d > 1e 1f 20 21 22 23 24 25 26 27 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 20 > 21 22 23 24 25 26 27 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23 > 24 25 26 27 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 > 26 27 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 > 26 27 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 27 > 28 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 2a > 2b > Mar 8 09:03:13 kaf-gimel last message repeated 34 times > Mar 8 09:03:13 kaf-gimel openais[3973]: [CPG ] got joinlist message > from node 1 > Mar 8 09:03:13 kaf-gimel openais[3973]: [CPG ] got joinlist message > from node 2 > Mar 8 09:03:13 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 34 > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] The token was lost in > the OPERATIONAL state. > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Receive multicast > socket recv buffer size (320000 bytes). > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Transmit multicast > socket send buffer size (262142 bytes). > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering GATHER state > from 2. > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Creating commit token > because I am the rep. > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Saving state aru 1c89 > high seq received 1c89 > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Storing new sequence > id for ring 48b88 > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering COMMIT > state. > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] entering RECOVERY > state. > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] position [0] member > 192.168.100.7: > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] previous ring seq > 297860 rep 192.168.100.7 > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] aru 1c89 high > delivered 1c89 received flag 1 > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] position [1] member > 192.168.100.10: > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] previous ring seq > 297860 rep 192.168.100.7 > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] aru 1c89 high > delivered 1c89 received flag 1 > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Did not need to > originate any messages in recovery. > Mar 8 09:04:41 kaf-gimel openais[3973]: [TOTEM] Sending initial ORF > token > Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION > CHANGE > Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] New Configuration: > Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.7) > Mar 8 09:04:41 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.10) > Mar 8 09:04:41 kaf-gimel gfs_controld[4004]: cpg_mcast_joined retry > 100 MSG_PLOCK > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Left: > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Joined: > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] CLM CONFIGURATION > CHANGE > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] New Configuration: > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.7) > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] r(0) > ip(192.168.100.10) > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Left: > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] Members Joined: > Mar 8 09:04:42 kaf-gimel openais[3973]: [SYNC ] This node is within > the primary component and will provide service. > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] entering OPERATIONAL > state. > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message > 192.168.100.7 > Mar 8 09:04:42 kaf-gimel openais[3973]: [CLM ] got nodejoin message > 192.168.100.10 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 12 > 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 23 > 24 25 26 27 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 > 1a 1b 1c 1d 1e 1f 20 21 23 24 25 26 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 1e > 1f 20 21 22 23 24 25 26 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 21 > 22 23 24 25 26 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 21 > 22 23 24 25 26 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 22 > 23 24 25 26 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 25 > 26 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 26 > 27 > Mar 8 09:04:42 kaf-gimel openais[3973]: [TOTEM] Retransmit List: 29 > 2a > Mar 8 09:04:42 kaf-gimel last message repeated 34 times > Mar 8 09:04:42 kaf-gimel openais[3973]: [CPG ] got joinlist message > from node 1 > Mar 8 09:04:42 kaf-gimel openais[3973]: [CPG ] got joinlist message > from node 2 > >
Are you using bonding? Regards -steve > _______________________________________________ > Openais mailing list > [email protected] > https://lists.linux-foundation.org/mailman/listinfo/openais _______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
