Re: [devel] [PATCH 0 of 1] Review Request for amf #601

Wed, 22 Jan 2014 23:50:32 -0800 

Attached is the corresponding 4.2/4.3 patch.
This patch lack the log change that was part of the 4.4 patch. That problem do exist in older branches as well but not for this use case. I think I should raise a minor defect for that one and handle it separately. Meaning you should disregard that part of the 4.4 patch since it will be handled by another ticket. 

Thanks,
Hans

On 01/10/2014 10:14 AM, Hans Feldt wrote:

Summary: Praveen
Review request for Trac Ticket(s): 601
Peer Reviewer(s): Praveen
Pull request to: <<LIST THE PERSON WITH PUSH ACCESS HERE>>
Affected branch(es): default
Development branch: <<IF ANY GIVE THE REPO URL>>

--------------------------------
Impacted area       Impact y/n
--------------------------------
  Docs                    n
  Build system            n
  RPM/packaging           n
  Configuration files     n
  Startup scripts         n
  SAF services            y
  OpenSAF services        n
  Core libraries          n
  Samples                 n
  Tests                   n
  Other                   n


Comments (indicate scope for each "y" above):
---------------------------------------------
This patch and its testing assumes a patch for #663 where both SU and node
admin ops are responded OK when an SU goes DISABLED.


changeset 82af3dd0b3026179ef1b7a07223af4f1e371e831
Author: Hans Feldt <[email protected]>
Date:   Fri, 10 Jan 2014 10:04:32 +0100

        amfnd: escalate comp errors in quiesced HA state [#601]

        When a node or SU is locked, AMF assigns the QUIESCED HA state to 
ACTIVE sa-
        aware components. This state is intended to transfer state to a peer
        component assigned STANDBY. If the ACTIVE component fails in the 
QUIESCED
        state, AMF will restart it and reassign it QUIESCED. This is an invalid
        state transition according to picture 3 in the B.04 AMF spec and makes 
no
        sense since the state is anyway gone (because the component failed).

        The problem was likely reintroduced when fixing #3083, before that 
special
        handling for comp errors in QUIESCED state was there. However the 
solution
        had other problems and was removed. So in the current code there is no
        special handling for errors in QUIESCED state which results in 
re-assignment
        of the QUIESCED HA state after component restart.

        This patch changes the error escalation logic so that instead of 
component
        restart, component failover is performed as recovery action.


Complete diffstat:
------------------
  osaf/services/saf/amf/amfnd/comp.cc             |  23 +++++++++++++++++++++--
  osaf/services/saf/amf/amfnd/err.cc              |  15 +++++++++++----
  osaf/services/saf/amf/amfnd/include/avnd_comp.h |   1 +
  osaf/services/saf/amf/amfnd/su.cc               |   1 +
  osaf/services/saf/amf/amfnd/util.cc             |   5 ++---
  5 files changed, 36 insertions(+), 9 deletions(-)


Testing Commands:
-----------------
slightly modified AppConfig-2N.xml from samples where SUs are pinned to nodes
amf_demo.c modified to exit when assigned QUIESCED

immcfg -f AppConfig-2N.xml
amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1
amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1
amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1
amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1
sleep 4
amf-adm lock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1
sleep 4
amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1
sleep 4
amf-adm lock safAmfNode=SC-2,safAmfCluster=myAmfCluster
sleep 4
amf-adm unlock safAmfNode=SC-2,safAmfCluster=myAmfCluster


Testing, Expected Results:
--------------------------
System stream:
         80 09:49:39 01/10/2014 NO safApp=safAmfService "Admin op "LOCK" 
initiated for 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1', invocation: 55834574849
         81 09:49:39 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState IN_SERVICE => 
OUT_OF_SERVICE
         82 09:49:39 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 AdmState UNLOCKED => LOCKED
         83 09:49:39 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 PresenceState TERMINATING => 
UNINSTANTIATED
         84 09:49:40 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 OperState ENABLED => DISABLED
         85 09:49:40 01/10/2014 NO safApp=safAmfService "Admin op done for 
invocation: 55834574849, result 1
         86 09:49:40 01/10/2014 NO safApp=safAmfService "Ordering Auto repair 
of 'safSg=AmfDemo,safApp=AmfDemo1' as sufailover repair action
         87 09:49:40 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 OperState DISABLED => ENABLED
         88 09:49:40 01/10/2014 NO safApp=safAmfService "HA State ACTIVE of 
safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1
         89 09:49:40 01/10/2014 NO safApp=safAmfService 
"safSi=AmfDemo,safApp=AmfDemo1 assigned to safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 
HA State UNKNOWN => ACTIVE
         90 09:49:40 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 PresenceState INSTANTIATING => 
INSTANTIATED
         91 09:49:44 01/10/2014 NO safApp=safAmfService "Admin op "UNLOCK" 
initiated for 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1', invocation: 60129542145
         92 09:49:44 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 AdmState LOCKED => UNLOCKED
         93 09:49:44 01/10/2014 NO safApp=safAmfService 
"safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState OUT_OF_SERVICE => 
IN_SERVICE
         94 09:49:44 01/10/2014 NO safApp=safAmfService "HA State STANDBY of 
safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1
         95 09:49:44 01/10/2014 NO safApp=safAmfService "Admin op done for 
invocation: 60129542145, result 1
         96 09:49:48 01/10/2014 NO safApp=safAmfService "Admin op "LOCK" 
initiated for 'safAmfNode=SC-2,safAmfCluster=myAmfCluster', invocation: 64424509441
         97 09:49:48 01/10/2014 NO safApp=safAmfService 
"safAmfNode=SC-2,safAmfCluster=myAmfCluster AdmState UNLOCKED => LOCKED
         98 09:49:48 01/10/2014 NO safApp=safAmfService 
"safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState IN_SERVICE => 
OUT_OF_SERVICE
         99 09:49:48 01/10/2014 NO safApp=safAmfService 
"safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 PresenceState TERMINATING => 
UNINSTANTIATED
        100 09:49:48 01/10/2014 NO safApp=safAmfService 
"safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 OperState ENABLED => DISABLED
        101 09:49:48 01/10/2014 NO safApp=safAmfService "Admin op done for 
invocation: 64424509441, result 1
        102 09:49:48 01/10/2014 NO safApp=safAmfService "Ordering Auto repair 
of 'safSg=AmfDemo,safApp=AmfDemo1' as sufailover repair action
        103 09:49:48 01/10/2014 NO safApp=safAmfService 
"safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 OperState DISABLED => ENABLED
        104 09:49:48 01/10/2014 NO safApp=safAmfService "HA State ACTIVE of 
safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1
        105 09:49:48 01/10/2014 NO safApp=safAmfService 
"safSi=AmfDemo,safApp=AmfDemo1 assigned to safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 
HA State UNKNOWN => ACTIVE
        106 09:49:48 01/10/2014 NO safApp=safAmfService 
"safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 PresenceState INSTANTIATING => 
INSTANTIATED
        107 09:49:52 01/10/2014 NO safApp=safAmfService "Admin op "UNLOCK" 
initiated for 'safAmfNode=SC-2,safAmfCluster=myAmfCluster', invocation: 68719476737
        108 09:49:52 01/10/2014 NO safApp=safAmfService 
"safAmfNode=SC-2,safAmfCluster=myAmfCluster AdmState LOCKED => UNLOCKED
        109 09:49:52 01/10/2014 NO safApp=safAmfService 
"safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState OUT_OF_SERVICE => 
IN_SERVICE
        110 09:49:52 01/10/2014 NO safApp=safAmfService "HA State STANDBY of 
safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1
        111 09:49:52 01/10/2014 NO safApp=safAmfService "Admin op done for 
invocation: 68719476737, result 1

SC-1 syslog (annotated, see inline):
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Assigned 
'safSi=AmfDemo,safApp=AmfDemo1' ACTIVE to 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Assigning 
'safSi=AmfDemo,safApp=AmfDemo1' QUIESCED to 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
Jan 10 09:39:28 SC-1 user.info amf_demo[654]: CSI Set - HAState Quiesced for 
all assigned CSIs
Jan 10 09:39:28 SC-1 user.err amf_demo[654]: QUIESCED exiting <========= fault 
injection
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO component with 
QUIESCED/QUIESCING assignment failed <===== new log
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO recovery action 'comp 
restart' escalated to 'comp failover' <===== new escalation
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO saAmfSUFailover is true 
for 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 
'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' faulted due to 
'avaDown' : Recovery is 'suFailover'
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Terminating components of 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'(abruptly & unordered)
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State INSTANTIATED => 
TERMINATING
Jan 10 09:39:28 SC-1 user.notice amf_demo_script: cleanup 
safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1, PID 654
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State TERMINATING => 
UNINSTANTIATED
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Terminated all components 
in 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Informing director of 
sufailover
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Repair request for 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' <===== new log
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State UNINSTANTIATED => 
UNINSTANTIATED
Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 
'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State UNINSTANTIATED => 
INSTANTIATING
Jan 10 09:39:28 SC-1 user.info amf_demo[692]: 
'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' started


Conditions of Submission:
-------------------------
  Ack from Praveen


Arch      Built     Started    Linux distro
-------------------------------------------
mips        n          n
mips64      n          n
x86         n          n
x86_64      y          y        ubuntu13.10 / UML
powerpc     n          n
powerpc64   n          n


Reviewer Checklist:
-------------------
[Submitters: make sure that your review doesn't trigger any checkmarks!]


Your checkin has not passed review because (see checked entries):

___ Your RR template is generally incomplete; it has too many blank entries
     that need proper data filled in.

___ You have failed to nominate the proper persons for review and push.

___ Your patches do not have proper short+long header

___ You have grammar/spelling in your header that is unacceptable.

___ You have exceeded a sensible line length in your headers/comments/text.

___ You have failed to put in a proper Trac Ticket # into your commits.

___ You have incorrectly put/left internal data in your comments/files
     (i.e. internal bug tracking tool IDs, product names etc)

___ You have not given any evidence of testing beyond basic build tests.
     Demonstrate some level of runtime or other sanity testing.

___ You have ^M present in some of your files. These have to be removed.

___ You have needlessly changed whitespace or added whitespace crimes
     like trailing spaces, or spaces before tabs.

___ You have mixed real technical changes with whitespace and other
     cosmetic code cleanup changes. These have to be separate commits.

___ You need to refactor your submission into logical chunks; there is
     too much content into a single commit.

___ You have extraneous garbage in your review (merge commits etc)

___ You have giant attachments which should never have been sent;
     Instead you should place your content in a public tree to be pulled.

___ You have too many commits attached to an e-mail; resend as threaded
     commits, or place in a public tree for a pull.

___ You have resent this content multiple times without a clear indication
     of what has changed between each re-send.

___ You have failed to adequately and individually address all of the
     comments and change requests that were proposed in the initial review.

___ You have a misconfigured ~/.hgrc file (i.e. username, email etc)

___ Your computer have a badly configured date and time; confusing the
     the threaded patch review.

___ Your changes affect IPC mechanism, and you don't present any results
     for in-service upgradability test.

___ Your changes affect user manual and documentation, your patch series
     do not contain the patch that updates the Doxygen manual.


------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-devel



# HG changeset patch
# Parent 9b9df21bd38203b0add0e99e10f8045459a66fd5
amfnd: escalate comp errors in quiesced HA state [#601]

When a node or SU is locked, AMF assigns the QUIESCED HA state to ACTIVE
sa-aware components. This state is intended to transfer state to a peer
component assigned STANDBY. If the ACTIVE component fails in the QUIESCED
state, AMF will restart it and reassign it QUIESCED. This is an invalid state
transition according to picture 3 in the B.04 AMF spec and makes no sense since
the state is anyway gone (because the component failed).

The problem was likely reintroduced when fixing #3083, before that special
handling for comp errors in QUIESCED state was there. However the solution had
other problems and was removed. So in the current code there is no special
handling for errors in QUIESCED state which results in re-assignment of the
QUIESCED HA state after component restart.

This patch changes the error escalation logic so that instead of component
restart, component failover is performed as recovery action.

diff --git a/osaf/services/saf/avsv/avnd/avnd_comp.c 
b/osaf/services/saf/avsv/avnd/avnd_comp.c
--- a/osaf/services/saf/avsv/avnd/avnd_comp.c
+++ b/osaf/services/saf/avsv/avnd/avnd_comp.c
@@ -2780,3 +2780,24 @@ void avnd_comp_pres_state_set(AVND_COMP 
        m_AVND_SEND_CKPT_UPDT_ASYNC_UPDT(cb, comp, AVND_CKPT_COMP_PRES_STATE);
 }
 
+/**
+ * Returns true if the HA state for any CSI assignment is QUIESCED/QUIESCING
+ * @param su
+ */
+bool comp_has_quiesced_assignment(const AVND_COMP *comp)
+{
+       const AVND_COMP_CSI_REC *csi;
+
+       for (csi = m_AVND_CSI_REC_FROM_COMP_DLL_NODE_GET(
+                       m_NCS_DBLIST_FIND_FIRST(&comp->csi_list));
+               csi != NULL;
+               csi = m_AVND_CSI_REC_FROM_COMP_DLL_NODE_GET(
+                       m_NCS_DBLIST_FIND_NEXT(&csi->comp_dll_node))) {
+
+               if ((csi->si->curr_state == SA_AMF_HA_QUIESCED) ||
+                               (csi->si->curr_state == SA_AMF_HA_QUIESCING))
+                       return true;
+       }
+
+       return false;
+}
diff --git a/osaf/services/saf/avsv/avnd/avnd_err.c 
b/osaf/services/saf/avsv/avnd/avnd_err.c
--- a/osaf/services/saf/avsv/avnd/avnd_err.c
+++ b/osaf/services/saf/avsv/avnd/avnd_err.c
@@ -411,9 +411,18 @@ uint32_t avnd_err_escalate(AVND_CB *cb, 
        if (*io_esc_rcvr == SA_AMF_NO_RECOMMENDATION)
                *io_esc_rcvr = comp->err_info.def_rec;
 
-       /* disallow comp-restart if it's disabled */
-       if ((SA_AMF_COMPONENT_RESTART == *io_esc_rcvr) && 
m_AVND_COMP_IS_RESTART_DIS(comp))
-               *io_esc_rcvr = AVSV_ERR_RCVR_SU_FAILOVER;
+       if (*io_esc_rcvr == SA_AMF_COMPONENT_RESTART) {
+               if (m_AVND_COMP_IS_RESTART_DIS(comp) && (!su->is_ncs)) {
+                       LOG_NO("saAmfCompDisableRestart is true for 
'%s'",comp->name.value);
+                       LOG_NO("recovery action 'comp restart' escalated to 
'comp failover'");
+                       *io_esc_rcvr = SA_AMF_COMPONENT_FAILOVER;
+               } else if (comp_has_quiesced_assignment(comp) == true) {
+                       /* Cannot re-assign QUIESCED, escalate to failover */
+                       LOG_NO("component with QUIESCED/QUIESCING assignment 
failed");
+                       LOG_NO("recovery action 'comp restart' escalated to 
'comp failover'");
+                       *io_esc_rcvr = SA_AMF_COMPONENT_FAILOVER;
+               }
+       }
 
        switch (*io_esc_rcvr) {
        case SA_AMF_COMPONENT_FAILOVER: /* treat it as su failover */
diff --git a/osaf/services/saf/avsv/avnd/include/avnd_comp.h 
b/osaf/services/saf/avsv/avnd/include/avnd_comp.h
--- a/osaf/services/saf/avsv/avnd/include/avnd_comp.h
+++ b/osaf/services/saf/avsv/avnd/include/avnd_comp.h
@@ -894,4 +894,6 @@ extern void avnd_comp_pres_state_set(AVN
  */
 extern uint32_t comp_restart_initiate(AVND_COMP *comp);
 
+bool comp_has_quiesced_assignment(const AVND_COMP *comp);
+
 #endif   /* !AVND_COMP_H */

------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today. 
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-devel






















					Reply via email to