Summary: amfd: reboot nodes that report conflicting 2N active assignments [#2920] Review request for Ticket(s): 2920 Peer Reviewer(s): *** LIST THE TECH REVIEWER(S) / MAINTAINER(S) HERE *** Pull request to: *** LIST THE PERSON WITH PUSH ACCESS HERE *** Affected branch(es): develop Development branch: ticket-2920 Base revision: 75342d931bb9d967048d350f6125169aca2e304c Personal repository: git://git.code.sf.net/u/userid-2226215/review
-------------------------------- Impacted area Impact y/n -------------------------------- Docs n Build system n RPM/packaging n Configuration files n Startup scripts n SAF services n OpenSAF services n Core libraries n Samples n Tests n Other n Comments (indicate scope for each "y" above): --------------------------------------------- revision 1cb5f00bb25729129ee7dc2edf11790a0debbd11 Author: Gary Lee <gary....@dektech.com.au> Date: Fri, 31 Aug 2018 06:37:03 +0000 amfd: reboot nodes that report conflicting 2N active assignments [#2920] After a split network event, both SCs can reboot endlessly, due to this assertion: 2018-08-29 18:05:34.689 SC-2 osafamfd[263]: src/amf/amfd/sg_2n_fsm.cc:596: avd_sg_2n_act_susi: Assertion 'a_susi_1->su == a_susi_2->su' failed. 2018-08-29 18:05:34.695 SC-2 osafamfnd[273]: ER AMFD has unexpectedly crashed. Rebooting node During the network split, a SC could assign another SU to be active, if the node hosting the old active 2N assignment is not reachable. The assert occurs after the network is merged. SC absence must be enabled. For now, we can aid recovery of the cluster by rebooting both of the PLs in place of the assertion. Complete diffstat: ------------------ src/amf/amfd/sg_2n_fsm.cc | 35 +++++++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) Testing Commands: ----------------- See ticket Testing, Expected Results: -------------------------- Instead of amfd asserting, we see: 2018-08-31 16:41:24.871 SC-1 osafamfd[272]: NO Cluster startup is done 2018-08-31 16:41:24.872 SC-1 osafamfd[272]: NO Perform absent failover for failed SU:safSu=1,safSg=1,safApp=AmfDemo 2018-08-31 16:41:24.873 SC-1 osafamfd[272]: EM Duplicate 2N active assignments in 'safSu=1,safSg=1,safApp=AmfDemo' and 'safSu=2,safSg=1,safApp=AmfDemo' 2018-08-31 16:41:24.873 SC-1 osafamfd[272]: EM Sending node reboot order to 'safAmfNode=PL-3,safAmfCluster=myAmfCluster' 2018-08-31 16:41:24.876 SC-1 osafamfd[272]: EM Sending node reboot order to 'safAmfNode=PL-4,safAmfCluster=myAmfCluster' Conditions of Submission: ------------------------- Ack from any reviewer Arch Built Started Linux distro ------------------------------------------- mips n n mips64 n n x86 n n x86_64 y y powerpc n n powerpc64 n n Reviewer Checklist: ------------------- [Submitters: make sure that your review doesn't trigger any checkmarks!] Your checkin has not passed review because (see checked entries): ___ Your RR template is generally incomplete; it has too many blank entries that need proper data filled in. ___ You have failed to nominate the proper persons for review and push. ___ Your patches do not have proper short+long header ___ You have grammar/spelling in your header that is unacceptable. ___ You have exceeded a sensible line length in your headers/comments/text. ___ You have failed to put in a proper Trac Ticket # into your commits. ___ You have incorrectly put/left internal data in your comments/files (i.e. internal bug tracking tool IDs, product names etc) ___ You have not given any evidence of testing beyond basic build tests. Demonstrate some level of runtime or other sanity testing. ___ You have ^M present in some of your files. These have to be removed. ___ You have needlessly changed whitespace or added whitespace crimes like trailing spaces, or spaces before tabs. ___ You have mixed real technical changes with whitespace and other cosmetic code cleanup changes. These have to be separate commits. ___ You need to refactor your submission into logical chunks; there is too much content into a single commit. ___ You have extraneous garbage in your review (merge commits etc) ___ You have giant attachments which should never have been sent; Instead you should place your content in a public tree to be pulled. ___ You have too many commits attached to an e-mail; resend as threaded commits, or place in a public tree for a pull. ___ You have resent this content multiple times without a clear indication of what has changed between each re-send. ___ You have failed to adequately and individually address all of the comments and change requests that were proposed in the initial review. ___ You have a misconfigured ~/.gitconfig file (i.e. user.name, user.email etc) ___ Your computer have a badly configured date and time; confusing the the threaded patch review. ___ Your changes affect IPC mechanism, and you don't present any results for in-service upgradability test. ___ Your changes affect user manual and documentation, your patch series do not contain the patch that updates the Doxygen manual. ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Opensaf-devel mailing list Opensaf-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/opensaf-devel