Hello, We found that sometimes Event Service Initialization fails due to unclear reason. It returns with service not available. Aisexec traces are attached below. Any clue why it happens?
Regards Ramesh Openais traces below. =============================== [MAIN ] AIS Executive Service RELEASE 'subrev 1152 version 0.80' [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. [MAIN ] Copyright (C) 2006 Red Hat, Inc. [MAIN ] AIS Executive Service: started and ready to provide service. [MAIN ] openais component openais_cpg loaded. [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' [MAIN ] openais component openais_cfg loaded. [MAIN ] Registering service handler 'openais configuration service' [MAIN ] openais component openais_msg loaded. [MAIN ] Registering service handler 'openais message service B.01.01' [MAIN ] openais component openais_lck loaded. [MAIN ] Registering service handler 'openais distributed locking service B.01.01' [MAIN ] openais component openais_evt loaded. [MAIN ] Registering service handler 'openais event service B.01.01' [MAIN ] openais component openais_ckpt loaded. [MAIN ] Registering service handler 'openais checkpoint service B.01.01' [MAIN ] openais component openais_amf loaded. [MAIN ] Registering service handler 'openais availability management framework B.01.01' [MAIN ] openais component openais_clm loaded. [MAIN ] Registering service handler 'openais cluster membership service B.01.01' [MAIN ] openais component openais_evs loaded. [MAIN ] Registering service handler 'openais extended virtual synchrony service' [TOTEM] Token Timeout (1000 ms) retransmit timeout (238 ms) [TOTEM] token hold (180 ms) retransmits before loss (4 retrans) [TOTEM] join (50 ms) send_join (0 ms) consensus (800 ms) merge (200 ms) [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) [TOTEM] send threads (0 threads) [TOTEM] RRP token expired timeout (238 ms) [TOTEM] RRP token problem counter (2000 ms) [TOTEM] RRP threshold (10 problem count) [TOTEM] RRP mode set to none. [TOTEM] heartbeat_failures_allowed (0) [TOTEM] max_network_delay (50 ms) [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 [TOTEM] Receive multicast socket recv buffer size (262142 bytes). [TOTEM] Transmit multicast socket send buffer size (262142 bytes). [TOTEM] The network interface [16.17.13.154] is now up. [TOTEM] Created or loaded sequence id 592.16.17.13.154 for this ring. [TOTEM] entering GATHER state from 15. [SERV ] Initialising service handler 'openais extended virtual synchrony service' [SERV ] Initialising service handler 'openais cluster membership service B.01.01' [SERV ] Initialising service handler 'openais availability management framework B.01.01' [SERV ] Initialising service handler 'openais checkpoint service B.01.01' [SERV ] Initialising service handler 'openais event service B.01.01' [SERV ] Initialising service handler 'openais distributed locking service B.01.01' [SERV ] Initialising service handler 'openais message service B.01.01' [SERV ] Initialising service handler 'openais configuration service' [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' [SYNC ] Not using a virtual synchrony filter. [MAIN ] ERROR: Could not bind AF_UNIX: Address already in use. [MAIN ] AIS Executive exiting (reason: could not bind to an address). [MAIN ] AIS Executive Service RELEASE 'subrev 1152 version 0.80' [MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors. [MAIN ] Copyright (C) 2006 Red Hat, Inc. [MAIN ] AIS Executive Service: started and ready to provide service. [MAIN ] openais component openais_cpg loaded. [MAIN ] Registering service handler 'openais cluster closed process group service v1.01' [MAIN ] openais component openais_cfg loaded. [MAIN ] Registering service handler 'openais configuration service' [MAIN ] openais component openais_msg loaded. [MAIN ] Registering service handler 'openais message service B.01.01' [MAIN ] openais component openais_lck loaded. [MAIN ] Registering service handler 'openais distributed locking service B.01.01' [MAIN ] openais component openais_evt loaded. [MAIN ] Registering service handler 'openais event service B.01.01' [MAIN ] openais component openais_ckpt loaded. [MAIN ] Registering service handler 'openais checkpoint service B.01.01' [MAIN ] openais component openais_amf loaded. [MAIN ] Registering service handler 'openais availability management framework B.01.01' [MAIN ] openais component openais_clm loaded. [MAIN ] Registering service handler 'openais cluster membership service B.01.01' [MAIN ] openais component openais_evs loaded. [MAIN ] Registering service handler 'openais extended virtual synchrony service' [TOTEM] Token Timeout (1000 ms) retransmit timeout (238 ms) [TOTEM] token hold (180 ms) retransmits before loss (4 retrans) [TOTEM] join (50 ms) send_join (0 ms) consensus (800 ms) merge (200 ms) [TOTEM] downcheck (1000 ms) fail to recv const (50 msgs) [TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500 [TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages) [TOTEM] send threads (0 threads) [TOTEM] RRP token expired timeout (238 ms) [TOTEM] RRP token problem counter (2000 ms) [TOTEM] RRP threshold (10 problem count) [TOTEM] RRP mode set to none. [TOTEM] heartbeat_failures_allowed (0) [TOTEM] max_network_delay (50 ms) [TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0 [TOTEM] Receive multicast socket recv buffer size (262142 bytes). [TOTEM] Transmit multicast socket send buffer size (262142 bytes). [TOTEM] The network interface [16.17.13.154] is now up. [TOTEM] Created or loaded sequence id 600.16.17.13.154 for this ring. [TOTEM] entering GATHER state from 15. [SERV ] Initialising service handler 'openais extended virtual synchrony service' [SERV ] Initialising service handler 'openais cluster membership service B.01.01' [SERV ] Initialising service handler 'openais availability management framework B.01.01' [SERV ] Initialising service handler 'openais checkpoint service B.01.01' [SERV ] Initialising service handler 'openais event service B.01.01' [SERV ] Initialising service handler 'openais distributed locking service B.01.01' [SERV ] Initialising service handler 'openais message service B.01.01' [SERV ] Initialising service handler 'openais configuration service' [SERV ] Initialising service handler 'openais cluster closed process group service v1.01' [SYNC ] Not using a virtual synchrony filter. [TOTEM] Creating commit token because I am the rep. [TOTEM] Saving state aru 0 high seq received 0 [TOTEM] Storing new sequence id for ring 25c [TOTEM] entering COMMIT state. [TOTEM] entering RECOVERY state. [TOTEM] position [0] member 16.17.13.154: [TOTEM] previous ring seq 600 rep 16.17.13.154 [TOTEM] aru 0 high delivered 0 received flag 1 [TOTEM] Did not need to originate any messages in recovery. [TOTEM] Sending initial ORF token [CLM ] CLM CONFIGURATION CHANGE [CLM ] New Configuration: [CLM ] Members Left: [CLM ] Members Joined: [CLM ] CLM CONFIGURATION CHANGE [CLM ] New Configuration: [CLM ] r(0) ip(16.17.13.154) [CLM ] Members Left: [CLM ] Members Joined: [CLM ] r(0) ip(16.17.13.154) [SYNC ] This node is within the primary component and will provide service. [TOTEM] entering OPERATIONAL state. [CLM ] got nodejoin message 16.17.13.154 [TOTEM] entering GATHER state from 11. [TOTEM] Saving state aru aaf7 high seq received aaf7 [TOTEM] Storing new sequence id for ring 260 [TOTEM] entering COMMIT state. [TOTEM] entering RECOVERY state. [TOTEM] position [0] member 16.17.12.55: [TOTEM] previous ring seq 588 rep 16.17.12.55 [TOTEM] aru 9 high delivered 9 received flag 1 [TOTEM] position [1] member 16.17.13.154: [TOTEM] previous ring seq 604 rep 16.17.13.154 [TOTEM] aru aaf7 high delivered aaf7 received flag 1 [TOTEM] Did not need to originate any messages in recovery. [CLM ] CLM CONFIGURATION CHANGE [CLM ] New Configuration: [CLM ] r(0) ip(16.17.13.154) [CLM ] Members Left: [CLM ] Members Joined: [CLM ] CLM CONFIGURATION CHANGE [CLM ] New Configuration: [CLM ] r(0) ip(16.17.12.55) [CLM ] r(0) ip(16.17.13.154) [CLM ] Members Left: [CLM ] Members Joined: [CLM ] r(0) ip(16.17.12.55) [SYNC ] This node is within the primary component and will provide service. [TOTEM] entering OPERATIONAL state. [CLM ] got nodejoin message 16.17.12.55 [CLM ] got nodejoin message 16.17.13.154 [EVT ] Channel NTF_CHANNEL_NAME, total 0, local 0 [EVT ] Node r(0) ip(16.17.13.154) , count 0 [TOTEM] Retransmit List: 7a 7b 7c 7d 7e [TOTEM] Retransmit List: 23d 23e 23f 240 [TOTEM] Retransmit List: 253 254 255 256 [TOTEM] Retransmit List: 268 269 [TOTEM] Retransmit List: 40e 40f [TOTEM] Retransmit List: 571 [TOTEM] Retransmit List: 588 [TOTEM] Retransmit List: 5bc [TOTEM] Retransmit List: 692 [TOTEM] Retransmit List: 753 [TOTEM] Retransmit List: 898 899 89a [TOTEM] Retransmit List: 9ba [TOTEM] Retransmit List: d22 d23 d24 [TOTEM] Retransmit List: d6c d6d d6e [TOTEM] Retransmit List: d6d d6e [TOTEM] Retransmit List: d9c d9d [TOTEM] Retransmit List: e00 [TOTEM] Retransmit List: e09 e0a [TOTEM] Retransmit List: faa fab [TOTEM] Retransmit List: 10a2 10a3 10a4 [TOTEM] Retransmit List: 1106 1107 1108 [TOTEM] Retransmit List: 1192 1193 1194 [TOTEM] Retransmit List: 122a [TOTEM] Retransmit List: 13f1 [TOTEM] Retransmit List: 1664 [TOTEM] Retransmit List: 1734 1735 1736 1737 [TOTEM] Retransmit List: 1756 [TOTEM] Retransmit List: 17a4 [TOTEM] Retransmit List: 1818 1819 181a [TOTEM] Retransmit List: 1851 1852 1853 1854 1855 1857 185a [TOTEM] Retransmit List: 1955 1956 1957 [TOTEM] Retransmit List: 19ce [TOTEM] Retransmit List: 1be9 1bea 1beb 1bec 1bed 1bef [TOTEM] Retransmit List: 1bf1 [TOTEM] Retransmit List: 1c6a 1c6b 1c6c 1c6d 1c6e 1c70 [TOTEM] Retransmit List: 1dfd 1dfe [TOTEM] Retransmit List: 1e0c 1e0d 1e0e 1e0f [TOTEM] Retransmit List: 1eed 1eee 1eef 1ef0 [TOTEM] Retransmit List: 1f14 1f15 [TOTEM] Retransmit List: 1f2d 1f2e 1f2f 1f30 1f31 1f32 [TOTEM] Retransmit List: 200c 200d 200e 200f 2010 2012 2014 2019 [TOTEM] Retransmit List: 20e3 [TOTEM] Retransmit List: 2106 [TOTEM] Retransmit List: 215a 215b [TOTEM] Retransmit List: 2166 2167 2168 2169 216a [TOTEM] Retransmit List: 2197 [TOTEM] Retransmit List: 21e0 21e1 21e2 21e3 21e4 21e5 21e6 [TOTEM] Retransmit List: 2240 2241 2242 2243 2244 2245 [TOTEM] Retransmit List: 2336 2337 2338 [TOTEM] Retransmit List: 2365 2366 2367 2368 2369 236a [TOTEM] Retransmit List: 236c [TOTEM] Retransmit List: 23d3 [TOTEM] Retransmit List: 2413 2414 2415 2416 2417 2418 [TOTEM] Retransmit List: 25c5 25c6 25c7 25c8 25c9 [TOTEM] Retransmit List: 2603 2604 2605 2606 2607 2608 2609 [TOTEM] Retransmit List: 26c4 [TOTEM] Retransmit List: 2765 2766 2767 2768 2769 276a 276b [TOTEM] FAILED TO RECEIVE [TOTEM] entering GATHER state from 6. [TOTEM] Saving state aru 2783 high seq received 2783 [TOTEM] Storing new sequence id for ring 264 [TOTEM] entering COMMIT state. [TOTEM] entering RECOVERY state. [TOTEM] position [0] member 16.17.12.55: [TOTEM] previous ring seq 608 rep 16.17.12.55 [TOTEM] aru 277e high delivered 277e received flag 1 [TOTEM] position [1] member 16.17.13.154: [TOTEM] previous ring seq 608 rep 16.17.12.55 [TOTEM] aru 2783 high delivered 2783 received flag 1 [TOTEM] Did not need to originate any messages in recovery. [CLM ] CLM CONFIGURATION CHANGE [CLM ] New Configuration: [CLM ] r(0) ip(16.17.12.55) [CLM ] r(0) ip(16.17.13.154) [CLM ] Members Left: [CLM ] Members Joined: [CLM ] CLM CONFIGURATION CHANGE [CLM ] New Configuration: [CLM ] r(0) ip(16.17.12.55) [CLM ] r(0) ip(16.17.13.154) [CLM ] Members Left: [CLM ] Members Joined: [SYNC ] This node is within the primary component and will provide service. [TOTEM] entering OPERATIONAL state. [CLM ] got nodejoin message 16.17.13.154 [CLM ] got nodejoin message 16.17.12.55 =========================================== Ramesh Hegde Software Engineer, Hewlett Packard India Software Operations Pvt. Ltd Sy No 192, Whitefield Road Mahadevapura Post Bangalore - 560 048. India. +91 80 2516 6486- Direct +91 80 2513 3522 - Fax Jabber Id : [EMAIL PROTECTED] +hp = everything is possible
_______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
