Summary: Praveen Review request for Trac Ticket(s): 601 Peer Reviewer(s): Praveen Pull request to: <<LIST THE PERSON WITH PUSH ACCESS HERE>> Affected branch(es): default Development branch: <<IF ANY GIVE THE REPO URL>>
-------------------------------- Impacted area Impact y/n -------------------------------- Docs n Build system n RPM/packaging n Configuration files n Startup scripts n SAF services y OpenSAF services n Core libraries n Samples n Tests n Other n Comments (indicate scope for each "y" above): --------------------------------------------- This patch and its testing assumes a patch for #663 where both SU and node admin ops are responded OK when an SU goes DISABLED. changeset 82af3dd0b3026179ef1b7a07223af4f1e371e831 Author: Hans Feldt <[email protected]> Date: Fri, 10 Jan 2014 10:04:32 +0100 amfnd: escalate comp errors in quiesced HA state [#601] When a node or SU is locked, AMF assigns the QUIESCED HA state to ACTIVE sa- aware components. This state is intended to transfer state to a peer component assigned STANDBY. If the ACTIVE component fails in the QUIESCED state, AMF will restart it and reassign it QUIESCED. This is an invalid state transition according to picture 3 in the B.04 AMF spec and makes no sense since the state is anyway gone (because the component failed). The problem was likely reintroduced when fixing #3083, before that special handling for comp errors in QUIESCED state was there. However the solution had other problems and was removed. So in the current code there is no special handling for errors in QUIESCED state which results in re-assignment of the QUIESCED HA state after component restart. This patch changes the error escalation logic so that instead of component restart, component failover is performed as recovery action. Complete diffstat: ------------------ osaf/services/saf/amf/amfnd/comp.cc | 23 +++++++++++++++++++++-- osaf/services/saf/amf/amfnd/err.cc | 15 +++++++++++---- osaf/services/saf/amf/amfnd/include/avnd_comp.h | 1 + osaf/services/saf/amf/amfnd/su.cc | 1 + osaf/services/saf/amf/amfnd/util.cc | 5 ++--- 5 files changed, 36 insertions(+), 9 deletions(-) Testing Commands: ----------------- slightly modified AppConfig-2N.xml from samples where SUs are pinned to nodes amf_demo.c modified to exit when assigned QUIESCED immcfg -f AppConfig-2N.xml amf-adm unlock-in safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock-in safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 amf-adm unlock safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 sleep 4 amf-adm lock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 sleep 4 amf-adm unlock safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 sleep 4 amf-adm lock safAmfNode=SC-2,safAmfCluster=myAmfCluster sleep 4 amf-adm unlock safAmfNode=SC-2,safAmfCluster=myAmfCluster Testing, Expected Results: -------------------------- System stream: 80 09:49:39 01/10/2014 NO safApp=safAmfService "Admin op "LOCK" initiated for 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1', invocation: 55834574849 81 09:49:39 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState IN_SERVICE => OUT_OF_SERVICE 82 09:49:39 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 AdmState UNLOCKED => LOCKED 83 09:49:39 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 PresenceState TERMINATING => UNINSTANTIATED 84 09:49:40 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 OperState ENABLED => DISABLED 85 09:49:40 01/10/2014 NO safApp=safAmfService "Admin op done for invocation: 55834574849, result 1 86 09:49:40 01/10/2014 NO safApp=safAmfService "Ordering Auto repair of 'safSg=AmfDemo,safApp=AmfDemo1' as sufailover repair action 87 09:49:40 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 OperState DISABLED => ENABLED 88 09:49:40 01/10/2014 NO safApp=safAmfService "HA State ACTIVE of safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1 89 09:49:40 01/10/2014 NO safApp=safAmfService "safSi=AmfDemo,safApp=AmfDemo1 assigned to safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 HA State UNKNOWN => ACTIVE 90 09:49:40 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 PresenceState INSTANTIATING => INSTANTIATED 91 09:49:44 01/10/2014 NO safApp=safAmfService "Admin op "UNLOCK" initiated for 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1', invocation: 60129542145 92 09:49:44 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 AdmState LOCKED => UNLOCKED 93 09:49:44 01/10/2014 NO safApp=safAmfService "safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState OUT_OF_SERVICE => IN_SERVICE 94 09:49:44 01/10/2014 NO safApp=safAmfService "HA State STANDBY of safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1 95 09:49:44 01/10/2014 NO safApp=safAmfService "Admin op done for invocation: 60129542145, result 1 96 09:49:48 01/10/2014 NO safApp=safAmfService "Admin op "LOCK" initiated for 'safAmfNode=SC-2,safAmfCluster=myAmfCluster', invocation: 64424509441 97 09:49:48 01/10/2014 NO safApp=safAmfService "safAmfNode=SC-2,safAmfCluster=myAmfCluster AdmState UNLOCKED => LOCKED 98 09:49:48 01/10/2014 NO safApp=safAmfService "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState IN_SERVICE => OUT_OF_SERVICE 99 09:49:48 01/10/2014 NO safApp=safAmfService "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 PresenceState TERMINATING => UNINSTANTIATED 100 09:49:48 01/10/2014 NO safApp=safAmfService "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 OperState ENABLED => DISABLED 101 09:49:48 01/10/2014 NO safApp=safAmfService "Admin op done for invocation: 64424509441, result 1 102 09:49:48 01/10/2014 NO safApp=safAmfService "Ordering Auto repair of 'safSg=AmfDemo,safApp=AmfDemo1' as sufailover repair action 103 09:49:48 01/10/2014 NO safApp=safAmfService "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 OperState DISABLED => ENABLED 104 09:49:48 01/10/2014 NO safApp=safAmfService "HA State ACTIVE of safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1 105 09:49:48 01/10/2014 NO safApp=safAmfService "safSi=AmfDemo,safApp=AmfDemo1 assigned to safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1 HA State UNKNOWN => ACTIVE 106 09:49:48 01/10/2014 NO safApp=safAmfService "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 PresenceState INSTANTIATING => INSTANTIATED 107 09:49:52 01/10/2014 NO safApp=safAmfService "Admin op "UNLOCK" initiated for 'safAmfNode=SC-2,safAmfCluster=myAmfCluster', invocation: 68719476737 108 09:49:52 01/10/2014 NO safApp=safAmfService "safAmfNode=SC-2,safAmfCluster=myAmfCluster AdmState LOCKED => UNLOCKED 109 09:49:52 01/10/2014 NO safApp=safAmfService "safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 ReadinessState OUT_OF_SERVICE => IN_SERVICE 110 09:49:52 01/10/2014 NO safApp=safAmfService "HA State STANDBY of safSu=SU2,safSg=AmfDemo,safApp=AmfDemo1 for safSi=AmfDemo,safApp=AmfDemo1 111 09:49:52 01/10/2014 NO safApp=safAmfService "Admin op done for invocation: 68719476737, result 1 SC-1 syslog (annotated, see inline): Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Assigned 'safSi=AmfDemo,safApp=AmfDemo1' ACTIVE to 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Assigning 'safSi=AmfDemo,safApp=AmfDemo1' QUIESCED to 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Jan 10 09:39:28 SC-1 user.info amf_demo[654]: CSI Set - HAState Quiesced for all assigned CSIs Jan 10 09:39:28 SC-1 user.err amf_demo[654]: QUIESCED exiting <========= fault injection Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO component with QUIESCED/QUIESCING assignment failed <===== new log Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO recovery action 'comp restart' escalated to 'comp failover' <===== new escalation Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO saAmfSUFailover is true for 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' faulted due to 'avaDown' : Recovery is 'suFailover' Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Terminating components of 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1'(abruptly & unordered) Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State INSTANTIATED => TERMINATING Jan 10 09:39:28 SC-1 user.notice amf_demo_script: cleanup safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1, PID 654 Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State TERMINATING => UNINSTANTIATED Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Terminated all components in 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Informing director of sufailover Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO Repair request for 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' <===== new log Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State UNINSTANTIATED => UNINSTANTIATED Jan 10 09:39:28 SC-1 local0.notice osafamfnd[447]: NO 'safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' Presence State UNINSTANTIATED => INSTANTIATING Jan 10 09:39:28 SC-1 user.info amf_demo[692]: 'safComp=AmfDemo,safSu=SU1,safSg=AmfDemo,safApp=AmfDemo1' started Conditions of Submission: ------------------------- Ack from Praveen Arch Built Started Linux distro ------------------------------------------- mips n n mips64 n n x86 n n x86_64 y y ubuntu13.10 / UML powerpc n n powerpc64 n n Reviewer Checklist: ------------------- [Submitters: make sure that your review doesn't trigger any checkmarks!] Your checkin has not passed review because (see checked entries): ___ Your RR template is generally incomplete; it has too many blank entries that need proper data filled in. ___ You have failed to nominate the proper persons for review and push. ___ Your patches do not have proper short+long header ___ You have grammar/spelling in your header that is unacceptable. ___ You have exceeded a sensible line length in your headers/comments/text. ___ You have failed to put in a proper Trac Ticket # into your commits. ___ You have incorrectly put/left internal data in your comments/files (i.e. internal bug tracking tool IDs, product names etc) ___ You have not given any evidence of testing beyond basic build tests. Demonstrate some level of runtime or other sanity testing. ___ You have ^M present in some of your files. These have to be removed. ___ You have needlessly changed whitespace or added whitespace crimes like trailing spaces, or spaces before tabs. ___ You have mixed real technical changes with whitespace and other cosmetic code cleanup changes. These have to be separate commits. ___ You need to refactor your submission into logical chunks; there is too much content into a single commit. ___ You have extraneous garbage in your review (merge commits etc) ___ You have giant attachments which should never have been sent; Instead you should place your content in a public tree to be pulled. ___ You have too many commits attached to an e-mail; resend as threaded commits, or place in a public tree for a pull. ___ You have resent this content multiple times without a clear indication of what has changed between each re-send. ___ You have failed to adequately and individually address all of the comments and change requests that were proposed in the initial review. ___ You have a misconfigured ~/.hgrc file (i.e. username, email etc) ___ Your computer have a badly configured date and time; confusing the the threaded patch review. ___ Your changes affect IPC mechanism, and you don't present any results for in-service upgradability test. ___ Your changes affect user manual and documentation, your patch series do not contain the patch that updates the Doxygen manual. ------------------------------------------------------------------------------ CenturyLink Cloud: The Leader in Enterprise Cloud Services. Learn Why More Businesses Are Choosing CenturyLink Cloud For Critical Workloads, Development Environments & Everything In Between. Get a Quote or Start a Free Trial Today. http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk _______________________________________________ Opensaf-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-devel
