Steve, I've been able to reproduce this reliably _without_ Pacemaker
being involved.
Attached are the two openais log files.
Scenario:
t0: hikari and hikari2 are up and can see each other
t1: Powercycle hikari2
t2: hikari2 comes up
t3: ping confirms that hikari2 can contact hikari
I modified the openais init script to run: ping -c 10 hikari >
afile
t4: hikari2 starts openais
t5: hikari starts producing membership events every 3s but does not
form a membership with hikari2
t6: hikari2 forms a membership by itself
t7: (about 1 or 2 minutes after t6, it varies) hikari and hikari2 form
a combined membership
The strangest part of this, is that hikari2 must reboot in order to
trigger this behavior.
Stopping or killing aisexec and then starting it again is not
sufficient.
Do you want to continue the discussion here or move to bugzilla?
Jul 30 05:40:58 hikari2 openais[4243]: [MAIN ] AIS Executive Service RELEASE
'subrev 1152 version 0.80'
Jul 30 05:40:58 hikari2 openais[4243]: [MAIN ] Copyright (C) 2002-2006
MontaVista Software, Inc and contributors.
Jul 30 05:40:58 hikari2 openais[4243]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Jul 30 05:40:58 hikari2 openais[4243]: [MAIN ] AIS Executive Service: started
and ready to provide service.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Token Timeout (20000 ms)
retransmit timeout (1960 ms)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] token hold (1558 ms) retransmits
before loss (10 retrans)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] join (1000 ms) send_join (45 ms)
consensus (2500 ms) merge (200 ms)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] downcheck (1000 ms) fail to recv
const (50 msgs)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] seqno unchanged const (30
rotations) Maximum network MTU 1500
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] window size per rotation (50
messages) maximum messages per rotation (20 messages)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] send threads (0 threads)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] RRP token expired timeout (1960
ms)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] RRP token problem counter (2000
ms)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] RRP threshold (10 problem count)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] RRP mode set to none.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] heartbeat_failures_allowed (0)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] max_network_delay (50 ms)
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] HeartBeat is Disabled. To enable
set heartbeat_failures_allowed > 0
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Receive multicast socket recv
buffer size (262142 bytes).
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Transmit multicast socket send
buffer size (262142 bytes).
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] The network interface
[172.16.1.112] is now up.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Created or loaded sequence id
147692.172.16.1.112 for this ring.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] entering GATHER state from 15.
Jul 30 05:40:58 hikari2 openais[4243]: [MAIN ] Service failed to load
'pacemake'.
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
extended virtual synchrony service'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
cluster membership service B.01.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
availability management framework B.01.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
checkpoint service B.01.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
event service B.01.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
distributed locking service B.01.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
message service B.01.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
configuration service'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
cluster closed process group service v1.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SERV ] Service initialized 'openais
cluster config database access v1.01'
Jul 30 05:40:58 hikari2 openais[4243]: [SYNC ] Not using a virtual synchrony
filter.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Creating commit token because I
am the rep.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Saving state aru 0 high seq
received 0
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Storing new sequence id for ring
240f0
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] entering COMMIT state.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] entering RECOVERY state.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] position [0] member 172.16.1.112:
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] previous ring seq 147692 rep
172.16.1.112
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] aru 0 high delivered 0 received
flag 1
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] Sending initial ORF token
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] New Configuration:
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] Members Left:
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] Members Joined:
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] New Configuration:
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] r(0) ip(172.16.1.112)
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] Members Left:
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] Members Joined:
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] r(0) ip(172.16.1.112)
Jul 30 05:40:58 hikari2 openais[4243]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:40:58 hikari2 openais[4243]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:40:58 hikari2 openais[4243]: [CLM ] got nodejoin message 172.16.1.112
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] entering GATHER state from 11.
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] Saving state aru a high seq
received a
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] Storing new sequence id for ring
24130
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] entering COMMIT state.
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] previous ring seq 147756 rep
172.16.1.110
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] position [1] member 172.16.1.112:
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] previous ring seq 147696 rep
172.16.1.112
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] aru a high delivered a received
flag 1
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] New Configuration:
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] r(0) ip(172.16.1.112)
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] Members Left:
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] Members Joined:
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] New Configuration:
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] r(0) ip(172.16.1.112)
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] Members Left:
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] Members Joined:
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:46 hikari2 openais[4243]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:46 hikari2 openais[4243]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:46 hikari2 openais[4243]: [CLM ] got nodejoin message 172.16.1.112
Jul 30 05:35:49 hikari openais[390]: [TOTEM] The token was lost in the
OPERATIONAL state.
Jul 30 05:35:49 hikari openais[390]: [TOTEM] Receive multicast socket recv
buffer size (262142 bytes).
Jul 30 05:35:49 hikari openais[390]: [TOTEM] Transmit multicast socket send
buffer size (262142 bytes).
Jul 30 05:35:49 hikari openais[390]: [TOTEM] entering GATHER state from 2.
Jul 30 05:35:52 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:35:52 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:35:52 hikari openais[390]: [TOTEM] Saving state aru 16 high seq
received 16
Jul 30 05:35:52 hikari openais[390]: [TOTEM] Storing new sequence id for ring
240f0
Jul 30 05:35:52 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:35:52 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:35:52 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:35:52 hikari openais[390]: [TOTEM] previous ring seq 147692 rep
172.16.1.110
Jul 30 05:35:52 hikari openais[390]: [TOTEM] aru 16 high delivered 16 received
flag 1
Jul 30 05:35:52 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:35:52 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:35:52 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:35:52 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:35:52 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:35:52 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:35:52 hikari openais[390]: [CLM ] r(0) ip(172.16.1.112)
Jul 30 05:35:52 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:35:52 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:35:52 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:35:52 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:35:52 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:35:52 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:35:52 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:35:52 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:35:52 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:40:58 hikari openais[390]: [TOTEM] entering GATHER state from 11.
Jul 30 05:41:00 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:00 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:00 hikari openais[390]: [TOTEM] Saving state aru c high seq
received c
Jul 30 05:41:00 hikari openais[390]: [TOTEM] Storing new sequence id for ring
240f4
Jul 30 05:41:00 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:00 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:00 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:00 hikari openais[390]: [TOTEM] previous ring seq 147696 rep
172.16.1.110
Jul 30 05:41:00 hikari openais[390]: [TOTEM] aru c high delivered c received
flag 1
Jul 30 05:41:00 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:00 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:00 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:00 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:00 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:00 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:00 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:00 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:00 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:00 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:00 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:00 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:00 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:00 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:00 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:01 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:04 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:04 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:04 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:04 hikari openais[390]: [TOTEM] Storing new sequence id for ring
240f8
Jul 30 05:41:04 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:04 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:04 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:04 hikari openais[390]: [TOTEM] previous ring seq 147700 rep
172.16.1.110
Jul 30 05:41:04 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:04 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:04 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:04 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:04 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:04 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:04 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:04 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:04 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:04 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:04 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:04 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:04 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:04 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:04 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:04 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:04 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:07 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:07 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:07 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:07 hikari openais[390]: [TOTEM] Storing new sequence id for ring
240fc
Jul 30 05:41:07 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:07 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:07 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:07 hikari openais[390]: [TOTEM] previous ring seq 147704 rep
172.16.1.110
Jul 30 05:41:07 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:07 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:07 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:07 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:07 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:07 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:07 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:07 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:07 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:07 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:07 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:07 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:07 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:07 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:07 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:07 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:08 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:10 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:10 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:10 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:10 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24100
Jul 30 05:41:10 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:10 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:10 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:10 hikari openais[390]: [TOTEM] previous ring seq 147708 rep
172.16.1.110
Jul 30 05:41:10 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:10 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:10 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:10 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:10 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:10 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:10 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:10 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:10 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:10 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:10 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:10 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:10 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:10 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:10 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:10 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:11 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:13 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:13 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:13 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:13 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24104
Jul 30 05:41:13 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:13 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:13 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:13 hikari openais[390]: [TOTEM] previous ring seq 147712 rep
172.16.1.110
Jul 30 05:41:13 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:13 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:13 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:13 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:13 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:13 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:13 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:13 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:13 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:13 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:13 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:13 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:13 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:13 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:13 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:13 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:14 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:16 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:16 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:16 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:16 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24108
Jul 30 05:41:16 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:16 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:16 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:16 hikari openais[390]: [TOTEM] previous ring seq 147716 rep
172.16.1.110
Jul 30 05:41:16 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:16 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:16 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:16 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:16 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:16 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:16 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:16 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:16 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:16 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:16 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:16 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:16 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:16 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:16 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:16 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:17 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:20 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:20 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:20 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:20 hikari openais[390]: [TOTEM] Storing new sequence id for ring
2410c
Jul 30 05:41:20 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:20 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:20 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:20 hikari openais[390]: [TOTEM] previous ring seq 147720 rep
172.16.1.110
Jul 30 05:41:20 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:20 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:20 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:20 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:20 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:20 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:20 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:20 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:20 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:20 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:20 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:20 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:20 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:20 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:20 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:20 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:20 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:23 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:23 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:23 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:23 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24110
Jul 30 05:41:23 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:23 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:23 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:23 hikari openais[390]: [TOTEM] previous ring seq 147724 rep
172.16.1.110
Jul 30 05:41:23 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:23 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:23 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:23 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:23 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:23 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:23 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:23 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:23 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:23 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:23 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:23 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:23 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:23 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:23 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:23 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:23 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:26 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:26 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:26 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:26 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24114
Jul 30 05:41:26 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:26 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:26 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:26 hikari openais[390]: [TOTEM] previous ring seq 147728 rep
172.16.1.110
Jul 30 05:41:26 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:26 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:26 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:26 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:26 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:26 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:26 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:26 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:26 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:26 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:26 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:26 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:26 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:26 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:26 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:26 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:26 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:29 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:29 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:29 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:29 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24118
Jul 30 05:41:29 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:29 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:29 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:29 hikari openais[390]: [TOTEM] previous ring seq 147732 rep
172.16.1.110
Jul 30 05:41:29 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:29 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:29 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:29 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:29 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:29 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:29 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:29 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:29 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:29 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:29 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:29 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:29 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:29 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:29 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:29 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:30 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:32 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:32 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:32 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:32 hikari openais[390]: [TOTEM] Storing new sequence id for ring
2411c
Jul 30 05:41:32 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:32 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:32 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:32 hikari openais[390]: [TOTEM] previous ring seq 147736 rep
172.16.1.110
Jul 30 05:41:32 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:32 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:32 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:32 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:32 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:32 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:32 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:32 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:32 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:32 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:32 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:32 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:32 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:32 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:32 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:32 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:33 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:35 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:35 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:35 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:35 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24120
Jul 30 05:41:35 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:35 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:35 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:35 hikari openais[390]: [TOTEM] previous ring seq 147740 rep
172.16.1.110
Jul 30 05:41:35 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:35 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:35 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:35 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:35 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:35 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:35 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:35 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:35 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:35 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:35 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:35 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:35 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:35 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:35 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:35 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:36 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:38 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:38 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:38 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:38 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24124
Jul 30 05:41:38 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:38 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:38 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:38 hikari openais[390]: [TOTEM] previous ring seq 147744 rep
172.16.1.110
Jul 30 05:41:38 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:38 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:38 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:38 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:38 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:38 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:38 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:38 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:38 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:38 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:38 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:38 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:38 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:38 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:38 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:38 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:39 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:42 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:42 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:42 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:42 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24128
Jul 30 05:41:42 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:42 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:42 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:42 hikari openais[390]: [TOTEM] previous ring seq 147748 rep
172.16.1.110
Jul 30 05:41:42 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:42 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:42 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:42 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:42 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:42 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:42 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:42 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:42 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:42 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:42 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:42 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:42 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:42 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:42 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:42 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:42 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:45 hikari openais[390]: [TOTEM] entering GATHER state from 0.
Jul 30 05:41:45 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:45 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:45 hikari openais[390]: [TOTEM] Storing new sequence id for ring
2412c
Jul 30 05:41:45 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:45 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:45 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:45 hikari openais[390]: [TOTEM] previous ring seq 147752 rep
172.16.1.110
Jul 30 05:41:45 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:45 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:45 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:45 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:45 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:45 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:45 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:45 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:45 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:45 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:45 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:45 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:45 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:45 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:45 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:45 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:45 hikari openais[390]: [TOTEM] entering GATHER state from 9.
Jul 30 05:41:46 hikari openais[390]: [TOTEM] Creating commit token because I am
the rep.
Jul 30 05:41:46 hikari openais[390]: [TOTEM] Saving state aru b high seq
received b
Jul 30 05:41:46 hikari openais[390]: [TOTEM] Storing new sequence id for ring
24130
Jul 30 05:41:46 hikari openais[390]: [TOTEM] entering COMMIT state.
Jul 30 05:41:46 hikari openais[390]: [TOTEM] entering RECOVERY state.
Jul 30 05:41:46 hikari openais[390]: [TOTEM] position [0] member 172.16.1.110:
Jul 30 05:41:46 hikari openais[390]: [TOTEM] previous ring seq 147756 rep
172.16.1.110
Jul 30 05:41:46 hikari openais[390]: [TOTEM] aru b high delivered b received
flag 1
Jul 30 05:41:46 hikari openais[390]: [TOTEM] position [1] member 172.16.1.112:
Jul 30 05:41:46 hikari openais[390]: [TOTEM] previous ring seq 147696 rep
172.16.1.112
Jul 30 05:41:46 hikari openais[390]: [TOTEM] aru a high delivered a received
flag 1
Jul 30 05:41:46 hikari openais[390]: [TOTEM] Did not need to originate any
messages in recovery.
Jul 30 05:41:46 hikari openais[390]: [TOTEM] Sending initial ORF token
Jul 30 05:41:46 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:46 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:46 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:46 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:46 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:46 hikari openais[390]: [CLM ] CLM CONFIGURATION CHANGE
Jul 30 05:41:46 hikari openais[390]: [CLM ] New Configuration:
Jul 30 05:41:46 hikari openais[390]: [CLM ] r(0) ip(172.16.1.110)
Jul 30 05:41:46 hikari openais[390]: [CLM ] r(0) ip(172.16.1.112)
Jul 30 05:41:46 hikari openais[390]: [CLM ] Members Left:
Jul 30 05:41:46 hikari openais[390]: [CLM ] Members Joined:
Jul 30 05:41:46 hikari openais[390]: [CLM ] r(0) ip(172.16.1.112)
Jul 30 05:41:46 hikari openais[390]: [SYNC ] This node is within the primary
component and will provide service.
Jul 30 05:41:46 hikari openais[390]: [TOTEM] entering OPERATIONAL state.
Jul 30 05:41:46 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.110
Jul 30 05:41:46 hikari openais[390]: [CLM ] got nodejoin message 172.16.1.112
On Jul 21, 2009, at 1:49 PM, Lars Marowsky-Bree wrote:
On 2009-06-30T12:27:33, Andrew Beekhof <[email protected]> wrote:
I'm working with a cluster that's having trouble reforming.
Before I explain, here is the totem section (which is the same on
both
nodes, except for the nodeid).
Hi all, Steven,
this problem persist. After a reboot, we sometimes see memberships not
reforming - for example, A B C D E, C & D reboot, we end up with A-B-E
and C-D or C / D by themselves or some other really weird membership.
The problem persist with latest whitetank. Occassionally it seems that
one of the dlm_controld processes seems to be hogging IPC (which seems
to be quite affecting the rest of the system), but this isn't always
the
case.
It is not always reproducible, and the symptoms are, well, weird.
Has anyone else ever seen this?
Regards,
Lars
--
Architect Storage/HA, OPS Engineering, Novell, Inc.
SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar
Wilde
-- Andrew
_______________________________________________
Openais mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/openais
