Hi Nagu, Praveen
From patch 09 to patch 14, they are fixes for bugs that you also need
on top of patches #4.
The problems you reported should not happen if you have them. They are
regardless whether we *reboot node if transient states* or *adjust
transient states* (delayed failover).
Patch 09 -> Return TRY_AGAIN for pg track start/stop in headless
Patch 10 -> Resend pg information to directors after headless
Patch 11 -> There are two fixes in this patch: (11_1) Fix mapping su,
and (11_2) fix amfnd coredump given that we allow comp/su failover
(patch #5). I split them
Patch 12 -> Do not disable healthy SU
Patch 13 -> It's for one payload limitation
Patch 14 -> It's for transient state at csi level, written on top of
patch #6.
So you also need patch 09, 10, 11_1, 12, 13 on top of patch #4, and they
need to be reviewed and pushed together with #1->#4 as well.
The patch #5 #6 #7 #8 are on different view from "immediate escalation"
and "reboot node if transient states".
We will look at your assignment_recovery.patch.
I also attach patch 1620_amfd_adjust_csi_V2.diff, which is to fix the
issue in TC #27, but it also depends on conclusion of how to deal with
transient states after headless.
Thanks,
Minh
On 03/03/16 02:12, Nagendra Kumar wrote:
#1 I have applied patches #1 to #4 only. With this patches(not having
patch #6), I thought to have passed most of the following tests, but
they got failed(Listed below).
I could not test other scenarios (including alarms and notifications),
because I haven’t applied patch #6. I think there should be a simple
patch replacing patch #6, which handles transient state as ‘reboot the
node‘ if Amf finds SUSI in transient state on that node.
I am attaching a concept patch(assignment_recovery.patch), which pass
some of the scenarios and we are testing and enhancing it.
As Praveen has suggested that we need to reboot the node which is
undergoing in transient state to make it simple.
This patch reduces complexity and maintainability.
So, ACK for patch #1-#4 along with the attached patch.
Please note that the attached patch has been created on patch #6 of
yours, so please apply #1 to #4 and then #6 and then the attached patch.
Currently the patch is for 2N red model. We are working to make for
Nway Act and No red model (and possibly for Nway and NpM), we will
publish it tomorrow.
TC #1:
Configuration(Comp recovery is comp failover, saAmfSutDefSUFailover as
false) and logs attached(TC 1) in the ticket.
1. Start SC-1, PL-3 and PL-4. SU1 Act on PL-3 and SU2 Standby on SC-2.
2. Stop SC-1 and kill demo. It goes for comp failover as configured.
Ideally, node should reboot.
3. Start SC-1. After cluster timer expires, PL-4 got the following
error messages:
Mar 2 08:01:15 PM_PL-4 osafamfnd[20050]: CR SU-SI record addition
failed, SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 :
SI=safSi=AmfDemo,safApp=AmfDemo1
Mar 2 08:01:15 PM_PL-4 osafamfnd[20050]: CR SU-SI record addition
failed, SU= safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 :
SI=safSi=AmfDemo1,safApp=AmfDemo1
There is no assignment given for SU1. SU2 has Standby assignments:
safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1
saAmfSISUHAState=STANDBY(2)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1,safApp=AmfDemo1
saAmfSISUHAState=STANDBY(2)
Other problems: a.) Further command for locking SU1/SU2 fails in SG
unstable error.
b.) Immlist if SU2 gives the below
result, Standby assignment it prints as 4, which is wrong:
saAmfSUNumCurrStandbySIs SA_UINT32_T 4 (0x4)
saAmfSUNumCurrActiveSIs SA_UINT32_T 0 (0x0)
c.) Even if SC-2 joins, and you do
failover/switchover of SC-1, still same as above.
TC #2: After execution of TC #1, stop PL-3. In worst case, SU2
assignment should change to Act, which is not happening. After
stopping of PL-4 also, the same problems as TC #1. logs attached(TC 2).
TC #3: After TC #2, start PL-3 and start SC-2.
SU1 is instantiated, but no assignment and the same
problem as above.
When stop PL-4, SU1 gets assignments, the following
logs comes at SC-2:
Mar 2 09:06:18 PM_SC-2 osafamfd[8518]: ER avd_ckpt_siass:
safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 safSi=AmfDemo,safApp=AmfDemo1
does not exist
Mar 2 09:06:18 PM_SC-2 osafamfd[8518]: ER avd_ckpt_siass:
safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 safSi=AmfDemo1,safApp=AmfDemo1
does not exist
Mar 2 09:06:21 PM_SC-2 kernel: [ 3290.784933] tipc: Resetting link
<1.1.2:eth0-1.1.4:eth0>, peer not responding
Mar 2 09:06:21 PM_SC-2 kernel: [ 3290.784947] tipc: Lost link
<1.1.2:eth0-1.1.4:eth0> on network plane A
Mar 2 09:06:21 PM_SC-2 kernel: [ 3290.784956] tipc: Lost contact with
<1.1.4>
Start PL-4, SU2 gets Standby assignments and everything works fine
after that.
TC #4: Similar problems exist in the following test cases:
a.)Configuration same as TC #1 except saAmfSutDefSUFailover as true.
After killing demo, PL-3 went for reboot.
But the problem is the same as shown in TC #1, TC #2
and TC #3.
b.) Configuration same as TC #1 except with saAmfCtDefRecoveryOnError
as 2 and saAmfCtDefDisableRestart as 1.
But the problem is the same as shown in TC #1, TC #2
and TC #3.
c.)Configuration same as TC #1 except with saAmfCtDefRecoveryOnError
as 2 and saAmfCtDefDisableRestart as 1 and saAmfSutDefSUFailover as 1.
After killing demo, PL-3 went for reboot.
But the problem is the same as shown in TC #1, TC #2
and TC #3.
TC #5: Configuration same as TC #1 except with
saAmfCtDefRecoveryOnError as 2. Configuration and logs(TC 5) attached
in ticket.
1. Start SC-1, PL-3 and PL-4. SU1 Act on PL-3 and SU2 Standby on SC-2.
2. Stop SC-1 and kill demo. It goes for comp restart as configured.
3. Start SC-1. After SC-1 comes up and before cluster timer expires,
stop PL-3:
Even if PL-3 is stopped(see below PL-3 is not available), SU1 is still
having Act assignment and SU2 is having Standby assignment:
PM_SC-1:/home/nagu/views/staging # amf-state siass
safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1,safApp=AmfDemo1
saAmfSISUHAState=STANDBY(2)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU2\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1
saAmfSISUHAState=STANDBY(2)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=PL-4\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed3,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo,safApp=AmfDemo1
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU1\,safSg=AmfDemo\,safApp=AmfDemo1,safSi=AmfDemo1,safApp=AmfDemo1
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
TC #6: After TC #5, start PL-3:
SU1 is not given any assignment (may be because it exists in Amfd db):
Mar 2 14:22:06 PM_PL-3 osafamfwd[8318]: Started
Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO
'safSu=PL-3,safSg=NoRed,safApp=OpenSAF' Presence State INSTANTIATING
=> INSTANTIATED
Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO Assigning
'safSi=NoRed2,safApp=OpenSAF' ACTIVE to
'safSu=PL-3,safSg=NoRed,safApp=OpenSAF'
Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO Assigned
'safSi=NoRed2,safApp=OpenSAF' ACTIVE to
'safSu=PL-3,safSg=NoRed,safApp=OpenSAF'
Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State
UNINSTANTIATED => INSTANTIATING
Mar 2 14:22:06 PM_PL-3 opensafd: OpenSAF(5.0.M0 - 7282:4fbffe857512:)
services successfully started
Mar 2 14:22:06 PM_PL-3 amf_demo[8337]:
'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' started
Mar 2 14:22:06 PM_PL-3 osafamfnd[8259]: NO
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State INSTANTIATING
=> INSTANTIATED
Mar 2 14:22:06 PM_PL-3 amf_demo[8337]: HC started with AMF
TC #7: After TC #6:
Lock SU1: Amfnd of PL-3 throws error:
Mar 2 14:23:57 PM_PL-3 osafamfnd[8259]: ER susi_assign_evh:
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' has no assignments
This is obvious because, Amfnd doesn’t have any assignment.
SU1 admin state is locked, but SUSI is being shown on SU1.
TC #8: After TC #7:
Lock SU1, it throws error:
Admin operation is already going on
(su'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1
TC #9: Same as TC #6 except Configure saAmfCtDefRecoveryOnError as
Node Switchover/Failover/Failfast.
The problem reported in TC #4 exists.
Thanks
-Nagu
> -----Original Message-----
> From: Minh Hon Chau [mailto:minh.c...@dektech.com.au]
> Sent: 25 February 2016 14:14
> To: hans.nordeb...@ericsson.com; gary....@dektech.com.au; Nagendra
> Kumar; Praveen Malviya; minh.c...@dektech.com.au
> Cc: opensaf-devel@lists.sourceforge.net
> Subject: [PATCH 01 of 15] amfd: Add support for cloud resilience at
common
> libs [#1620]
amf: Fix mapping SU [#1620]
diff --git a/osaf/services/saf/amf/amfd/role.cc b/osaf/services/saf/amf/amfd/role.cc
--- a/osaf/services/saf/amf/amfd/role.cc
+++ b/osaf/services/saf/amf/amfd/role.cc
@@ -180,6 +180,11 @@
goto done;
}
+ if (avd_imm_impl_set() != SA_AIS_OK) {
+ LOG_ER("avd_imm_impl_set FAILED");
+ goto done;
+ }
+
if (avd_imm_config_get() != NCSCC_RC_SUCCESS) {
LOG_ER("avd_imm_config_get FAILED");
goto done;
@@ -187,11 +192,6 @@
cb->init_state = AVD_CFG_DONE;
- if (avd_imm_impl_set() != SA_AIS_OK) {
- LOG_ER("avd_imm_impl_set FAILED");
- goto done;
- }
-
avd_imm_update_runtime_attrs();
status = NCSCC_RC_SUCCESS;
diff --git a/osaf/services/saf/amf/amfd/siass.cc b/osaf/services/saf/amf/amfd/siass.cc
--- a/osaf/services/saf/amf/amfd/siass.cc
+++ b/osaf/services/saf/amf/amfd/siass.cc
@@ -888,7 +888,9 @@
susi->su->inc_curr_act_si();
susi->si->inc_curr_act_ass();
}
-
+ su->saAmfSUHostedByNode = node->name;
+ avd_saImmOiRtObjectUpdate(&su->name, "saAmfSUHostedByNode",
+ SA_IMM_ATTR_SANAMET, &su->saAmfSUHostedByNode);
m_AVSV_SEND_CKPT_UPDT_ASYNC_ADD(avd_cb, susi, AVSV_CKPT_AVD_SI_ASS);
}
diff --git a/osaf/services/saf/amf/amfd/cluster.cc b/osaf/services/saf/amf/amfd/cluster.cc
--- a/osaf/services/saf/amf/amfd/cluster.cc
+++ b/osaf/services/saf/amf/amfd/cluster.cc
@@ -98,6 +98,11 @@ void avd_cluster_tmr_init_evh(AVD_CL_CB
*/
if (cb->scs_absence_max_duration > 0) {
i_sg->adjust_intermediate_sg(cb);
+ /*
+ * bring up possible csi assignments before
+ * adjust si ha assignment in delayed_failover
+ */
+ i_sg->adjust_csi_assignment(cb);
i_sg->adjust_delayed_failover(cb);
}
if (i_sg->sg_fsm_state == AVD_SG_FSM_STABLE)
diff --git a/osaf/services/saf/amf/amfd/csi.cc b/osaf/services/saf/amf/amfd/csi.cc
--- a/osaf/services/saf/amf/amfd/csi.cc
+++ b/osaf/services/saf/amf/amfd/csi.cc
@@ -1527,3 +1527,90 @@ SaAisErrorT avd_compcsi_recreate(AVSV_N2
return SA_AIS_OK;
}
+/**
+ * @brief Adjust csi assignment after headless. Currently this function
+ * just adding more compcsi
+ *
+ * @param[in] csi pointer.
+ *
+ * @return None
+ */
+void AVD_CSI::adjust_assignment(AVD_CL_CB *cb)
+{
+ AVD_COMP *t_comp;
+ AVD_SU_SI_REL *t_sisu;
+ AVD_COMP_CSI_REL *compcsi;
+ TRACE_ENTER();
+
+ // Check whether csi assignment is already in progress and if yes, then return.
+ if (si->list_of_sisu != nullptr) {
+ for(t_sisu = si->list_of_sisu; t_sisu != nullptr; t_sisu = t_sisu->si_next) {
+ if (t_sisu->csi_add_rem == true) {
+ LOG_NO("CSI create '%s' delayed: pending assignment for '%s'",
+ name.value, t_sisu->su->name.value);
+ goto done;
+ }
+ }
+ }
+
+ // Check whether si has been assigned to any SU.
+ // Only susi doesn't has csi creation in progress
+ if (si->list_of_sisu != nullptr) {
+ t_sisu = si->list_of_sisu;
+ for (t_sisu = si->list_of_sisu; t_sisu; t_sisu = t_sisu->si_next) {
+
+ // Find unassigned comp
+ t_sisu->su->reset_all_comps_assign_flag();
+ compcsi = t_sisu->list_of_csicomp;
+ while (compcsi != nullptr) {
+ compcsi->comp->set_assigned(true);
+ compcsi = compcsi->susi_csicomp_next;
+ }
+ t_comp = t_sisu->su->find_unassigned_comp_that_provides_cstype(&saAmfCSType);
+
+ // No unassigned comp, find the assigned comp
+ if (t_comp == nullptr) {
+ t_comp = AVD_CSI::find_assigned_comp(&saAmfCSType, t_sisu, t_sisu->su->list_of_comp);
+ }
+
+ if (t_comp == nullptr) {
+ LOG_ER("Compcsi doesn't exist or MaxActiveCSI/MaxStandbyCSI have reached for csi '%s'",
+ name.value);
+ continue;
+ }
+
+ // Found right comp, don't create the existed one
+ bool compcsi_exist = false;
+ for (compcsi = t_sisu->list_of_csicomp; compcsi && !compcsi_exist;
+ compcsi = compcsi->susi_csicomp_next) {
+ if ((compcsi->comp == t_comp) && (compcsi->csi == this))
+ compcsi_exist = true;
+ }
+ if (compcsi_exist) {
+ TRACE("existed compcsi: %s, %s", name.value, t_comp->comp_info.name.value);
+ continue;
+ }
+ if ((compcsi = avd_compcsi_create(t_sisu, this, t_comp, true)) == nullptr) {
+ /* free all the CSI assignments and end this loop */
+ avd_compcsi_delete(cb, t_sisu, true);
+ break;
+ }
+ // Send susi_msg to create single csi
+ // Mark the SUSI record is being add_remove
+ if (t_sisu->csi_add_rem == false && t_sisu->fsm == AVD_SU_SI_STATE_ASGND) {
+ if (avd_snd_susi_msg(cb, t_sisu->su, t_sisu, AVSV_SUSI_ACT_ASGN, true, compcsi) != NCSCC_RC_SUCCESS) {
+ avd_compcsi_delete(cb, t_sisu, true);
+ avd_susi_update_assignment_counters(t_sisu, AVSV_SUSI_ACT_DEL, static_cast<SaAmfHAStateT>(0), static_cast<SaAmfHAStateT>(0));
+ avd_susi_delete(cb, t_sisu, true);
+ break;
+ }
+ t_sisu->csi_add_rem = static_cast<SaBoolT>(true);
+ t_sisu->comp_name = compcsi->comp->comp_info.name;
+ t_sisu->csi_name = compcsi->csi->name;
+ m_AVSV_SEND_CKPT_UPDT_ASYNC_UPDT(cb, t_sisu, AVSV_CKPT_AVD_SI_ASS);
+ }
+ }
+ }
+done:
+ TRACE_LEAVE();
+}
diff --git a/osaf/services/saf/amf/amfd/include/csi.h b/osaf/services/saf/amf/amfd/include/csi.h
--- a/osaf/services/saf/amf/amfd/include/csi.h
+++ b/osaf/services/saf/amf/amfd/include/csi.h
@@ -86,7 +86,7 @@ class AVD_CSI {
from * current SI being assigned */
static AVD_COMP* find_assigned_comp(const SaNameT *cstype, const AVD_SU_SI_REL *sisu, const std::vector<AVD_COMP*> &list_of_comp);
-
+ void adjust_assignment(AVD_CL_CB *cb);
private:
AVD_CSI();
// disallow copy and assign
diff --git a/osaf/services/saf/amf/amfd/include/sg.h b/osaf/services/saf/amf/amfd/include/sg.h
--- a/osaf/services/saf/amf/amfd/include/sg.h
+++ b/osaf/services/saf/amf/amfd/include/sg.h
@@ -425,6 +425,7 @@ public:
SaAisErrorT check_sg_stability();
bool ng_using_saAmfSGAdminState;
void adjust_intermediate_sg(AVD_CL_CB *cb);
+ void adjust_csi_assignment(AVD_CL_CB *cb);
uint32_t term_su_list_in_reverse();
//Runtime calculates value of saAmfSGNumCurrAssignedSUs;
uint32_t curr_assigned_sus() const;
diff --git a/osaf/services/saf/amf/amfd/sg.cc b/osaf/services/saf/amf/amfd/sg.cc
--- a/osaf/services/saf/amf/amfd/sg.cc
+++ b/osaf/services/saf/amf/amfd/sg.cc
@@ -2182,3 +2182,16 @@ void AVD_SG::adjust_intermediate_sg(AVD_
TRACE_LEAVE();
}
+void AVD_SG::adjust_csi_assignment(AVD_CL_CB *cb)
+{
+ TRACE_ENTER();
+ // csi adjustment for every csi in si
+ for (const auto& si : list_of_si) {
+ AVD_CSI* curr_csi;
+ for (curr_csi = si->list_of_csi; curr_csi;
+ curr_csi = curr_csi->si_list_of_csi_next) {
+ curr_csi->adjust_assignment(cb);
+ }
+ }
+ TRACE_LEAVE();
+}
amfnd: Return TRY_AGAIN for saAmfProtectionGroupTrack and saAmfProtectionGroupTrackStop [#1620]
Patch returns TRY_AGAIN for saAmfProtectionGroupTrack and saAmfProtectionGroupTrackStop
during headless since the proctection group tracking requires amfd's presence
diff --git a/osaf/services/saf/amf/amfnd/pg.cc b/osaf/services/saf/amf/amfnd/pg.cc
--- a/osaf/services/saf/amf/amfnd/pg.cc
+++ b/osaf/services/saf/amf/amfnd/pg.cc
@@ -147,6 +147,15 @@
TRACE_ENTER();
+ // if headless, return TRY_AGAIN to application
+ if (cb->is_avd_down == true) {
+ LOG_NO("Director is down. Return try again for PG start.");
+ rc = avnd_amf_resp_send(cb, AVSV_AMF_PG_START, SA_AIS_ERR_TRY_AGAIN,
+ 0, &api_info->dest, &evt->mds_ctxt, nullptr, false);
+ TRACE_LEAVE();
+ return rc;
+ }
+
/*
* Update pg db
*/
@@ -235,6 +244,15 @@
TRACE_ENTER();
+ // if headless, return TRY_AGAIN to application
+ if (cb->is_avd_down == true) {
+ LOG_NO("Director is down. Return try again for PG stop.");
+ rc = avnd_amf_resp_send(cb, AVSV_AMF_PG_STOP, SA_AIS_ERR_TRY_AGAIN,
+ 0, &api_info->dest, &evt->mds_ctxt, nullptr, false);
+ TRACE_LEAVE();
+ return rc;
+ }
+
/* populate the track key */
key.mds_dest = api_info->dest;
key.req_hdl = pg_stop->hdl;
amfnd: Resend pg information after headless [#1620]
If SC comes back from headless, currently protection group information
will be lost at amfd.
Patch resends protection group information, which is similiar to failover
diff --git a/osaf/services/saf/amf/amfnd/di.cc b/osaf/services/saf/amf/amfnd/di.cc
--- a/osaf/services/saf/amf/amfnd/di.cc
+++ b/osaf/services/saf/amf/amfnd/di.cc
@@ -1260,6 +1260,8 @@ void avnd_diq_rec_del(AVND_CB *cb, AVND_
avnd_diq_rec_send(cb, pending_rec);
}
}
+ /* resend pg start track */
+ avnd_di_resend_pg_start_track(cb);
}
/* free the avnd message contents */
@@ -1459,6 +1461,40 @@ uint32_t avnd_evt_avd_role_change_evh(AV
return rc;
}
+/****************************************************************************
+ Name : avnd_di_resend_pg_start_track
+
+ Description : This routing will get called on AVD fail-over or coming back
+ from headless to send the PG start messages to the new AVD.
+
+ Arguments : cb - ptr to the AvND control block
+
+ Return Values : NCSCC_RC_SUCCESS/NCSCC_RC_FAILURE
+
+ Notes : None.
+******************************************************************************/
+uint32_t avnd_di_resend_pg_start_track(AVND_CB *cb)
+{
+ uint32_t rc = NCSCC_RC_SUCCESS;
+ AVND_PG *pg = 0;
+ SaNameT csi_name;
+ TRACE_ENTER();
+
+ memset(&csi_name, '\0', sizeof(SaNameT));
+
+ while (nullptr != (pg = m_AVND_PGDB_REC_GET_NEXT(cb->pgdb, csi_name))) {
+ rc = avnd_di_pg_act_send(cb, &pg->csi_name, AVSV_PG_TRACK_ACT_START, true);
+
+ if (NCSCC_RC_SUCCESS != rc)
+ break;
+
+ csi_name = pg->csi_name;
+ }
+
+ TRACE_LEAVE();
+ return rc;
+}
+
/**
* The SC absence timer expired. Reboot this node.
* @param cb
diff --git a/osaf/services/saf/amf/amfnd/include/avnd_di.h b/osaf/services/saf/amf/amfnd/include/avnd_di.h
--- a/osaf/services/saf/amf/amfnd/include/avnd_di.h
+++ b/osaf/services/saf/amf/amfnd/include/avnd_di.h
@@ -83,6 +83,7 @@ uint32_t avnd_diq_rec_send(struct avnd_c
uint32_t avnd_di_reg_su_rsp_snd(struct avnd_cb_tag *cb, SaNameT *su_name, uint32_t ret_code);
uint32_t avnd_di_ack_nack_msg_send(struct avnd_cb_tag *cb, uint32_t rcv_id, uint32_t view_num);
extern void avnd_di_uns32_upd_send(int class_id, int attr_id, const SaNameT *dn, uint32_t value);
+extern uint32_t avnd_di_resend_pg_start_track(struct avnd_cb_tag *);
void avnd_sync_sisu(struct avnd_cb_tag *cb);
void avnd_sync_csicomp(struct avnd_cb_tag *cb);
diff --git a/osaf/services/saf/amf/amfnd/verify.cc b/osaf/services/saf/amf/amfnd/verify.cc
--- a/osaf/services/saf/amf/amfnd/verify.cc
+++ b/osaf/services/saf/amf/amfnd/verify.cc
@@ -34,42 +34,6 @@
#include "avnd.h"
-static uint32_t avnd_send_pg_start_on_fover(AVND_CB *cb);
-
-/****************************************************************************
- Name : avnd_send_pg_start_on_fover
-
- Description : This routing will get called on AVD fail-over to send the
- PG start messages to the new AVD.
-
- Arguments : cb - ptr to the AvND control block
-
- Return Values : NCSCC_RC_SUCCESS/NCSCC_RC_FAILURE
-
- Notes : None.
-******************************************************************************/
-static uint32_t avnd_send_pg_start_on_fover(AVND_CB *cb)
-{
- uint32_t rc = NCSCC_RC_SUCCESS;
- AVND_PG *pg = 0;
- SaNameT csi_name;
- TRACE_ENTER();
-
- memset(&csi_name, '\0', sizeof(SaNameT));
-
- while (nullptr != (pg = m_AVND_PGDB_REC_GET_NEXT(cb->pgdb, csi_name))) {
- rc = avnd_di_pg_act_send(cb, &pg->csi_name, AVSV_PG_TRACK_ACT_START, true);
-
- if (NCSCC_RC_SUCCESS != rc)
- break;
-
- csi_name = pg->csi_name;
- }
-
- TRACE_LEAVE();
- return rc;
-}
-
/****************************************************************************
Name : avnd_evt_avd_verify_message
@@ -173,7 +137,7 @@ uint32_t avnd_evt_avd_verify_evh(AVND_CB
/*
* Send PG tracking (START) message to new Active.
*/
- avnd_send_pg_start_on_fover(cb);
+ avnd_di_resend_pg_start_track(cb);
TRACE_LEAVE();
# HG changeset patch
# User Hans Nordeback <hans.nordeb...@ericsson.com>
# Date 1456144854 -3600
# Mon Feb 22 13:40:54 2016 +0100
# Node ID 76cc1a6bb7f2670a14ed24590a1b6b537626355d
# Parent 743bffd74700c2174a4af2f3ac0bbfa1de246f69
amfd: Reboot cluster at data inconsistency [#1620]
Cluster is preferly configured with one payload without PBE.
After two times of headless, IMM will reload from xml.
That cause amfd lost all objects which were created before headless
and the data inconsistency happens between amfnd and amfnd/IMM
The patch broadcast reboot message to all nodes
diff --git a/osaf/services/saf/amf/amfd/include/util.h b/osaf/services/saf/amf/amfd/include/util.h
--- a/osaf/services/saf/amf/amfd/include/util.h
+++ b/osaf/services/saf/amf/amfd/include/util.h
@@ -74,6 +74,7 @@ uint32_t avd_snd_node_up_msg(struct cl_c
uint32_t avd_snd_presence_msg(struct cl_cb_tag *cb, AVD_SU *su, bool term_state);
uint32_t avd_snd_oper_state_msg(struct cl_cb_tag *cb, AVD_AVND *avnd, uint32_t msg_id_ack);
uint32_t avd_snd_op_req_msg(struct cl_cb_tag *cb, AVD_AVND *avnd, AVSV_PARAM_INFO *param_info);
+void avd_d2n_snd_reboot_req_msg();
uint32_t avd_snd_su_reg_msg(struct cl_cb_tag *cb, AVD_AVND *avnd, bool fail_over);
uint32_t avd_snd_su_msg(struct cl_cb_tag *cb, AVD_SU *su);
uint32_t avd_snd_susi_msg(struct cl_cb_tag *cb, AVD_SU *su, struct avd_su_si_rel_tag *susi,
diff --git a/osaf/services/saf/amf/amfd/siass.cc b/osaf/services/saf/amf/amfd/siass.cc
--- a/osaf/services/saf/amf/amfd/siass.cc
+++ b/osaf/services/saf/amf/amfd/siass.cc
@@ -839,7 +839,14 @@ SaAisErrorT avd_susi_recreate(AVSV_N2D_N
su_state = su_state->next) {
AVD_SU *su = su_db->find(Amf::to_string(&su_state->safSU));
- osafassert(su);
+ if (su == nullptr) {
+ LOG_ER("SU data inconsistency detected. Ordering cluster reboot");
+ avd_d2n_snd_reboot_req_msg();
+ for (;;) {
+ LOG_ER("Waiting for reboot");
+ sleep(1);
+ }
+ }
// present state
su->set_pres_state(static_cast<SaAmfPresenceStateT>(su_state->su_pres_state));
diff --git a/osaf/services/saf/amf/amfd/util.cc b/osaf/services/saf/amf/amfd/util.cc
--- a/osaf/services/saf/amf/amfd/util.cc
+++ b/osaf/services/saf/amf/amfd/util.cc
@@ -1916,3 +1916,25 @@ bool admin_op_is_valid(SaImmAdminOperati
child_dn->length = i;
return 0;
}
+
+ /**
+ * Broadcasts a reboot request to all amf node directors.
+ * Use broadcast as adests may not be available at the time of reboot
+ * request.
+ */
+void avd_d2n_snd_reboot_req_msg() {
+ TRACE_ENTER();
+
+ AVD_DND_MSG *reboot_req_msg = new AVSV_DND_MSG();
+
+ /* prepare the reboot request message. */
+ reboot_req_msg->msg_type = AVSV_D2N_REBOOT_MSG;
+ reboot_req_msg->msg_info.d2n_reboot_info.msg_id = 0;
+
+ /* Broadcast the operation request message to all the nodes. */
+ avd_d2n_msg_bcast(avd_cb, reboot_req_msg);
+
+ delete reboot_req_msg;
+
+ TRACE_LEAVE();
+}
\ No newline at end of file
diff --git a/osaf/services/saf/amf/amfnd/amfnd.cc b/osaf/services/saf/amf/amfnd/amfnd.cc
--- a/osaf/services/saf/amf/amfnd/amfnd.cc
+++ b/osaf/services/saf/amf/amfnd/amfnd.cc
@@ -386,7 +386,8 @@ uint32_t avnd_evt_avd_reboot_evh(AVND_CB
osafassert(AVSV_D2N_REBOOT_MSG == evt->info.avd->msg_type);
- avnd_msgid_assert(info->msg_id);
+ if (info->msg_id)
+ avnd_msgid_assert(info->msg_id);
cb->rcv_msg_id = info->msg_id;
/* Clear error report related alarms before reboot.
diff --git a/osaf/services/saf/amf/amfnd/mds.cc b/osaf/services/saf/amf/amfnd/mds.cc
--- a/osaf/services/saf/amf/amfnd/mds.cc
+++ b/osaf/services/saf/amf/amfnd/mds.cc
@@ -337,7 +337,8 @@ uint32_t avnd_mds_rcv(AVND_CB *cb, MDS_C
* message, to the Anchor of the received message.
*/
if ((AVSV_D2N_NODE_UP_MSG == ((AVSV_DND_MSG *)(rcv_info->i_msg))->msg_type) ||
- (AVSV_D2N_DATA_VERIFY_MSG == ((AVSV_DND_MSG *)(rcv_info->i_msg))->msg_type)) {
+ (AVSV_D2N_DATA_VERIFY_MSG == ((AVSV_DND_MSG *)(rcv_info->i_msg))->msg_type) ||
+ (AVSV_D2N_REBOOT_MSG == ((AVSV_DND_MSG *)(rcv_info->i_msg))->msg_type)) {
cb->active_avd_adest = rcv_info->i_fr_dest;
TRACE_1("Active AVD Adest = %" PRIu64 ,cb->active_avd_adest);
}
amfd: Don't disable healthy SU [#1620]
This scenario happen if unlock-in SU before going headless.
After headless, amfnd sends SU oper state DISABLE in recovery data.
The patch comments out the suspicious setting SU's oper state to DISABLED
while knowing that SU is not FAILED.
diff --git a/osaf/services/saf/amf/amfnd/su.cc b/osaf/services/saf/amf/amfnd/su.cc
--- a/osaf/services/saf/amf/amfnd/su.cc
+++ b/osaf/services/saf/amf/amfnd/su.cc
@@ -574,9 +574,11 @@ uint32_t avnd_su_curr_info_del(AVND_CB *
/* stop su_err_esc_tmr TBD Later */
/* disable the oper state (if pi su) */
- if (m_AVND_SU_IS_PREINSTANTIABLE(su) && (su->admin_op_Id != SA_AMF_ADMIN_RESTART)) {
- m_AVND_SU_OPER_STATE_SET(su, SA_AMF_OPERATIONAL_DISABLED);
- }
+ // TODO: reset healthy SU to DISABLED doesn't seem to be right, double check
+ // how SU Operational state being set in other places
+ //if (m_AVND_SU_IS_PREINSTANTIABLE(su) && (su->admin_op_Id != SA_AMF_ADMIN_RESTART)) {
+ // m_AVND_SU_OPER_STATE_SET(su, SA_AMF_OPERATIONAL_DISABLED);
+ //}
}
/* scan & delete the current info store in each component */
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
