Hi AMF maintainers,
While waiting Mahesh checks whether another NCSMDS_DOWN(Vdest) should
come 3 mins after headless, can we have a look at this patch?
I think we need it to make AMFND safe.
Thanks,
Minh
On 27/04/17 12:26, A V Mahesh wrote:
> Hi Minh chau,
>
> On 4/26/2017 5:43 PM, minh chau wrote:
>>
>> - Stop both SCs, amfnd receives 2 NCSMDS_DOWN, one is Adest, one is
>> Vdest
>
> I don't seen unnatural events from MDS, as amfnd might have subsided
> for them.
> Currently transport (MDS) functionality doesn't provide event
> differently for
> headless or non-headless and it is completely invisible to MDS.
>
> I will go through this AMF case and will get back to you.
>
> -AVM
>
> On 4/26/2017 5:43 PM, minh chau wrote:
>> Hi Mahesh,
>>
>> The sequence is going like this:
>>
>> - Stop both SCs, amfnd receives 2 NCSMDS_DOWN, one is Adest, one is
>> Vdest. I guess at this point MDS tells that both standby and active
>> amfd are down?
>> 2017-04-26 21:13:52 PL-4 osafamfnd[413]: WA AMF director
>> unexpectedly crashed
>>
>> - Leave cluster in headless about 3 mins, amfnd receives another
>> NCSMDS_DOWN with Vdest, so MDS is telling no active amfd again?
>> syslog:
>> 2017-04-26 21:16:52 PL-4 osafamfnd[413]: WA AMF director
>> unexpectedly crashed
>>
>> mds log:
>> <143>1 2017-04-26T21:16:52.873168+10:00 PL-4 osafamfnd 413
>> mds.log [meta sequenceId="9881"] >> mds_mcm_await_active_tmr_expiry
>> <142>1 2017-04-26T21:16:52.873183+10:00 PL-4 osafamfnd 413
>> mds.log [meta sequenceId="9882"] MCM:API: await_active_tmr expired
>> for svc_id = AVND(13) Subscribed to svc_id = AVD(12) on VDEST id = 1
>> <143>1 2017-04-26T21:16:52.9453+10:00 PL-4 osafclmna 405 mds.log
>> [meta sequenceId="938"] >> mds_mcm_await_active_tmr_expiry
>> <142>1 2017-04-26T21:16:52.945309+10:00 PL-4 osafclmna 405
>> mds.log [meta sequenceId="939"] MCM:API: await_active_tmr expired for
>> svc_id = CLMNA(36) Subscribed to svc_id = CLMS(34) on VDEST id = 16
>> <142>1 2017-04-26T21:16:52.945452+10:00 PL-4 osafsmfnd 454
>> mds.log [meta sequenceId="620"] MCM:API: svc_down :
>> await_active_tmr_expiry : svc_id = SMFND(31) on DEST id = 65535 got
>> DOWN for svc_id = SMFD(30) on VDEST id = 15
>> <143>1 2017-04-26T21:16:52.945462+10:00 PL-4 osafsmfnd 454
>> mds.log [meta sequenceId="621"] << mds_mcm_await_active_tmr_expiry
>> <143>1 2017-04-26T21:16:52.945938+10:00 PL-4 osafckptnd 432
>> mds.log [meta sequenceId="1547"] >> mds_mcm_await_active_tmr_expiry
>> <142>1 2017-04-26T21:16:52.945947+10:00 PL-4 osafckptnd 432
>> mds.log [meta sequenceId="1548"] MCM:API: await_active_tmr expired
>> for svc_id = CPND(17) Subscribed to svc_id = CPD(16) on VDEST id = 9
>> <142>1 2017-04-26T21:16:52.946064+10:00 PL-4 osafckptnd 432
>> mds.log [meta sequenceId="1558"] MCM:API: svc_down :
>> await_active_tmr_expiry : svc_id = CPND(17) on DEST id = 65535 got
>> DOWN for svc_id = CPD(16) on VDEST id = 9
>> <143>1 2017-04-26T21:16:52.946074+10:00 PL-4 osafckptnd 432
>> mds.log [meta sequenceId="1559"] << mds_mcm_await_active_tmr_expiry
>> <143>1 2017-04-26T21:16:52.94611+10:00 PL-4 osafckptnd 432
>> mds.log [meta sequenceId="1562"] >> mds_mcm_await_active_tmr_expiry
>> <142>1 2017-04-26T21:16:52.946118+10:00 PL-4 osafckptnd 432
>> mds.log [meta sequenceId="1563"] MCM:API: await_active_tmr expired
>> for svc_id = CLMA(35) Subscribed to svc_id = CLMS(34) on VDEST id = 16
>> <143>1 2017-04-26T21:16:52.955692+10:00 PL-4 osafimmnd 395
>> mds.log [meta sequenceId="30048"] >> mds_mcm_await_active_tmr_expiry
>> <142>1 2017-04-26T21:16:52.955698+10:00 PL-4 osafimmnd 395
>> mds.log [meta sequenceId="30049"] MCM:API: await_active_tmr expired
>> for svc_id = CLMA(35) Subscribed to svc_id = CLMS(34) on VDEST id = 16
>> <142>1 2017-04-26T21:16:52.955765+10:00 PL-4 osafimmnd 395
>> mds.log [meta sequenceId="30059"] MCM:API: svc_down :
>> await_active_tmr_expiry : svc_id = CLMA(35) on DEST id = 65535 got
>> DOWN for svc_id = CLMS(34) on VDEST id = 16
>> <143>1 2017-04-26T21:16:52.955775+10:00 PL-4 osafimmnd 395
>> mds.log [meta sequenceId="30060"] << mds_mcm_await_active_tmr_expiry
>>
>> I guess the other node-director services also receive the 2nd
>> NCSMDS_DOWN(Vdest), but those services have no problem because of
>> service's logic (or likely ckptnd checks cb->is_cpd_up == true), so I
>> thought it would be AMF problem, until I see the points from
>> Suryanarayana. So the await_active_tmr is working as expected?
>>
>> thanks,
>> Minh
>>
>> On 26/04/17 17:11, A V Mahesh wrote:
>>> Hi Minh Chau,
>>>
>>> On 4/26/2017 12:05 PM, minh chau wrote:
>>>> amfnd will receive another NCSMDS_DOWN
>>>
>>> you mean amfnd is receiving NCSMDS_DOWN for same amfd twice ?
>>> or amfnd is receiving NCSMDS_DOWN for both active amfd & standby
>>> amfd ?
>>>
>>> -AVM
>>>
>>> On 4/26/2017 12:05 PM, minh chau wrote:
>>>>
>>>> @Suryanarayana: I think this fix makes AMFND a bit defensive, but
>>>> let's see Mahesh's comments
>>>> @Mahesh: If getting NCSMDS_DOWN, then there's no active to wait, so
>>>> MDS should stop this timer?
>>>>
>>>>
>>>> On 26/04/17 15:45, Suryanarayana.Garlapati wrote:
>>>>> Might be i guess this fix needs to be done at the MDS level, not
>>>>> at the AMFND, taking into consideration that the cluster
>>>>>
>>>>> has only two Controllers.
>>>>>
>>>>> Timer which is getting started at MDS should not be started(if
>>>>> started should be stopped) in case of getting the down for both of
>>>>> the amfd's.
>>>>>
>>>>>
>>>>>
>>>>> On Wednesday 26 April 2017 10:53 AM, Minh Chau wrote:
>>>>>> If cluster goes into headless stage and wait up to 3 mins
>>>>>> which is currently the timeout of MDS_AWAIT_ACTIVE_TMR_VAL,
>>>>>> amfnd will receive another NCSMDS_DOWN, and then delete
>>>>>> all buffered messages. As a result, the headless recovery
>>>>>> is impossible because these buffered messages are deleted.
>>>>>>
>>>>>> Patch ignores the second NCSMDS_DOWN.
>>>>>> ---
>>>>>> src/amf/amfnd/di.cc | 7 +++++++
>>>>>> 1 file changed, 7 insertions(+)
>>>>>>
>>>>>> diff --git a/src/amf/amfnd/di.cc b/src/amf/amfnd/di.cc
>>>>>> index 627b31853..e06b9260d 100644
>>>>>> --- a/src/amf/amfnd/di.cc
>>>>>> +++ b/src/amf/amfnd/di.cc
>>>>>> @@ -638,6 +638,13 @@ uint32_t avnd_evt_mds_avd_dn_evh(AVND_CB
>>>>>> *cb, AVND_EVT *evt) {
>>>>>> }
>>>>>> }
>>>>>> + // Ignore the second NCSMDS_DOWN which comes from timeout of
>>>>>> + // MDS_AWAIT_ACTIVE_TMR_VAL
>>>>>> + if (cb->is_avd_down == true) {
>>>>>> + TRACE_LEAVE();
>>>>>> + return rc;
>>>>>> + }
>>>>>> +
>>>>>> m_AVND_CB_AVD_UP_RESET(cb);
>>>>>> cb->active_avd_adest = 0;
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Opensaf-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
