2 other runs, with and without headless feature, test from cluster start up
Run 3: with headless feature
2016-11-16 22:31:14 SC-2 osafimmnd[429]: exiting for shutdown
2016-11-16 22:31:14 SC-2 osafimmd[419]: NO New IMMND process is on ACTIVE
Controller at 2020f
2016-11-16 22:31:14 SC-2 osafimmd[419]: NO MDS event from svc_id 25 (change:4,
dest:565213401186733)
2016-11-16 22:31:14 SC-2 osafntfimcnd[485]: NO saImmOiDispatch() Fail
SA_AIS_ERR_BAD_HANDLE (9)
2016-11-16 22:31:14 SC-2 osafamfnd[475]: NO saClmDispatch BAD_HANDLE
2016-11-16 22:31:14 SC-2 osafamfnd[475]: NO Starting hb supervision of local avd
2016-11-16 22:31:14 SC-2 osafimmd[419]: NO Sc Absence Allowed is configured
(900) => IMMND coord at payload node:2040f dest567412424442252
2016-11-16 22:31:14 SC-2 osafimmd[419]: NO Node 2010f request sync sync-pid:431
epoch:0
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO Successfully announced sync. New
ruling epoch:23
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO ACT: New Epoch for IMMND process at
node 2010f old epoch: 0 new epoch:23
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO ACT: New Epoch for IMMND process at
node 2030f old epoch: 22 new epoch:23
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO ACT: New Epoch for IMMND process at
node 2040f old epoch: 22 new epoch:23
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO ACT: SaImmRepositoryInitModeT
changed and noted as being: SA_IMM_KEEP_REPOSITORY
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO Successfully announced dump at node
2040f. New Epoch:24
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO ACT: New Epoch for IMMND process at
node 2030f old epoch: 23 new epoch:24
2016-11-16 22:31:15 SC-2 osafimmd[419]: NO ACT: New Epoch for IMMND process at
node 2010f old epoch: 23 new epoch:24
TEST: SC failover
2016-11-16 22:31:17 SC-2 osafdtmd[381]: NO Lost contact with 'SC-1'
2016-11-16 22:31:17 SC-2 osafimmd[419]: NO MDS event from svc_id 25 (change:4,
dest:564113889558959)
2016-11-16 22:31:17 SC-2 osafclmd[457]: NO Node 131343 went down. Not sending
track callback for agents on that node
2016-11-16 22:31:44 SC-2 osafamfnd[475]: CR saImmOmInitialize failed. Use
previous value of nodeName.
2016-11-16 22:34:13 SC-2 osafamfd[466]: ER saImmOmSearchInitialize_2 failed: 6
2016-11-16 22:34:13 SC-2 osafamfd[466]: ER Failed to read configuration, AMF
will not start
2016-11-16 22:34:13 SC-2 osafamfd[466]: ER avd_imm_config_get FAILED
2016-11-16 22:34:13 SC-2 osafamfd[466]: ER avd_active_role_initialization FAILED
2016-11-16 22:34:13 SC-2 osafamfd[466]: ER initialize_for_assignment FAILED 2
Run 4: Without headless feature
2016-11-16 22:38:33 SC-2 osafdtmd[383]: NO Lost contact with 'SC-1'
2016-11-16 22:38:33 SC-2 osafamfnd[477]: WA AMF director unexpectedly crashed
2016-11-16 22:38:33 SC-2 osafamfnd[477]: Rebooting OpenSAF NodeId = 131599 EE
Name = , Reason: local AVD down(Adest) or both AVD down(Vdest) received,
OwnNodeId = 131599, SupervisionTime = 60
2016-11-16 22:38:33 SC-2 osafimmnd[431]: ER No IMMD service => cluster restart,
exiting
2016-11-16 22:38:33 SC-2 opensaf_reboot: Rebooting local node; timeout=60
Then SC2 is coming from reboot
2016-11-16 22:38:41 SC-2 osafimmnd[434]: exiting for shutdown
2016-11-16 22:38:41 SC-2 osafimmd[424]: NO MDS event from svc_id 25 (change:4,
dest:565213401186738)
2016-11-16 22:38:41 SC-2 osafimmd[424]: WA IMMND coordinator at 2020f
apparently crashed => electing new coord
2016-11-16 22:38:41 SC-2 osafimmd[424]: ER Failed to find candidate for new
IMMND coordinator (ScAbsenceAllowed:0 RulingEpoch:28
2016-11-16 22:38:41 SC-2 osafimmd[424]: ER Active IMMD has to restart the
IMMSv. All IMMNDs will restart
2016-11-16 22:38:41 SC-2 osafimmd[424]: ER IMM RELOAD => ensure cluster
restart by IMMD exit at both SCs, exiting
2016-11-16 22:38:41 SC-2 osafntfimcnd[473]: NO saImmOiDispatch() Fail
SA_AIS_ERR_BAD_HANDLE (9)
2016-11-16 22:41:50 SC-2 osafamfd[489]: ER saImmOmSearchInitialize_2 failed: 6
2016-11-16 22:41:50 SC-2 osafamfd[489]: ER Failed to read configuration, AMF
will not start
2016-11-16 22:41:50 SC-2 osafamfd[489]: ER avd_imm_config_get FAILED
2016-11-16 22:41:50 SC-2 osafamfd[489]: ER avd_active_role_initialization FAILED
2016-11-16 22:41:50 SC-2 osafamfd[489]: ER initialize_for_assignment FAILED 2
2016-11-16 22:41:50 SC-2 opensafd[378]: ER Failed #012 DESC:AMFD
2016-11-16 22:41:50 SC-2 opensafd[378]: ER Going for recovery
2016-11-16 22:41:50 SC-2 osafamfd[489]: ER initialize failed, exiting
2016-11-16 22:41:50 SC-2 osafclmd[480]: exiting for shutdown
2016-11-16 22:41:50 SC-2 osafclmna[399]: exiting for shutdown
In 4 runs, the common issue is that IMMND could not be restarted since IMMND
has not been an AMF component.
---
** [tickets:#2158] AMF: IMMND dies at Opensaf start up phase causes AMFD
heartbeat timeout**
**Status:** unassigned
**Milestone:** 5.0.2
**Created:** Wed Nov 02, 2016 05:20 AM UTC by Minh Hon Chau
**Last Updated:** Wed Nov 16, 2016 10:52 AM UTC
**Owner:** nobody
**Attachments:**
-
[osafamfnd_sc2](https://sourceforge.net/p/opensaf/tickets/2158/attachment/osafamfnd_sc2)
(264.2 kB; application/octet-stream)
If IMMND dies at Opensaf startup phase, IMMND is not restarted by AMF. The
issue has been observed in following situation
- Restart cluster
- During active controller starts up, a critical component is death which cause
a node failfast
Oct 25 12:51:21 SC-1 osafamfnd[7642]: ER
safComp=ABC,safSu=1,safSg=2N,safApp=ABC Faulted due to:csiSetcallbackTimeout
Recovery is:nodeFailfast
Oct 25 12:51:21 SC-1 osafamfnd[7642]: Rebooting OpenSAF NodeId = 131343 EE Name
= , Reason: Component faulted: recovery is node failfast, OwnNodeId = 131343,
SupervisionTime = 60
- In the meantime, standby controller is requested to become active
Oct 25 12:51:27 SC-2 tipclog[16221]: Lost link <1.1.2:eth0-1.1.1:eth0> on
network plane A
Oct 25 12:51:27 SC-2 osafclmna[4336]: NO Starting to promote this node to a
system controller
Oct 25 12:51:27 SC-2 osafrded[4387]: NO Requesting ACTIVE role
- IMMND is also death a bit later
Oct 25 12:51:29 SC-2 osafimmnd[4536]: ER MESSAGE:44816 OUT OF ORDER my highest
processed:44814 - exiting
Oct 25 12:51:29 SC-2 osafamfnd[7414]: NO saClmDispatch BAD_HANDLE
- Other services could not initialize other services since IMMND is death
Oct 25 12:51:39 SC-2 osafamfd[7400]: WA saClmInitialize_4 returned 5
Oct 25 12:51:39 SC-2 osafamfd[7400]: WA saNtfInitialize returned 5
Oct 25 12:51:39 SC-2 osafntfimcnd[7501]: WA ntfimcn_ntf_init saNtfInitialize(
returned SA_AIS_ERR_TIMEOUT (5)
Oct 25 12:51:39 SC-2 osafclmd[7386]: WA saImmOiImplementerSet returned 9
Oct 25 12:51:39 SC-2 osafntfd[7372]: WA saLogInitialize returns try again,
retries...
Oct 25 12:51:39 SC-2 osaflogd[7358]: WA saImmOiImplementerSet returned
SA_AIS_ERR_BAD_HANDLE (9)
Oct 25 12:51:39 SC-2 osafamfnd[7414]: WA saClmInitialize_4 returned 5
Oct 25 12:51:49 SC-2 osafamfd[7400]: WA saClmInitialize_4 returned 5
Oct 25 12:51:50 SC-2 osafamfd[7400]: WA saNtfInitialize returned 5
Oct 25 12:51:50 SC-2 osafamfnd[7414]: WA saClmInitialize_4 returned 5
Oct 25 12:52:00 SC-2 osafamfd[7400]: WA saClmInitialize_4 returned 5
Oct 25 12:52:00 SC-2 osafamfd[7400]: WA saNtfInitialize returned 5
Oct 25 12:52:00 SC-2 osafamfnd[7414]: WA saClmInitialize_4 returned 5
Oct 25 12:52:20 SC-2 osafamfnd[7414]: WA saClmInitialize_4 returned 5
Oct 25 12:52:20 SC-2 osafamfd[7400]: WA saNtfInitialize returned 5
Oct 25 12:52:20 SC-2 osafimmd[4489]: NO Extended intro from node 2210f
- At the end, AMFD heart beat timeout
Oct 25 12:53:57 SC-2 osafntfimcnd[7501]: WA ntfimcn_ntf_init saNtfInitialize(
returned SA_AIS_ERR_TIMEOUT (5)
Oct 25 12:54:01 SC-2 osafamfnd[7414]: WA saClmInitialize_4 returned 5
Oct 25 12:54:01 SC-2 osafamfd[7400]: WA saNtfInitialize returned 5
Oct 25 12:54:01 SC-2 osafamfd[7400]: WA saClmInitialize_4 returned 5
Oct 25 12:54:07 SC-2 osafntfimcnd[7501]: WA ntfimcn_ntf_init saNtfInitialize(
returned SA_AIS_ERR_TIMEOUT (5)
Oct 25 12:54:11 SC-2 osafamfnd[7414]: WA saClmInitialize_4 returned 5
Oct 25 12:54:11 SC-2 osafamfd[7400]: WA saClmInitialize_4 returned 5
Oct 25 12:54:11 SC-2 osafamfd[7400]: WA saNtfInitialize returned 5
Oct 25 12:54:15 SC-2 osafamfnd[7414]: ER AMF director heart beat timeout,
generating core for amfd
In AMFND trace in SC2, AMFND did not receive su_pres from AMFD, therefore AMFND
could not initiate middleware components (including IMMND), so AMFND was not
aware of IMMND's death so that AMFND can restart IMMND. The problem here is
slightly different from #1828, which happened in newly promoted SC (with
roamingSC feature) where AMFND had IMMND registered.
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
_______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
