For the SMS healthcheck, The certificate had expired on may 31st. I have requested a new one and will add it to the repo as soon as I get it.
-- K i r a n From: [email protected] [mailto:[email protected]] On Behalf Of Morgan Richomme via Lists.Onap.Org Sent: Monday, June 03, 2019 9:29 AM To: FREEMAN, BRIAN D <[email protected]>; [email protected]; PLATANIA, MARCO <[email protected]>; [email protected] Cc: [email protected]; [email protected]; [email protected]; [email protected]; LEFEVRE, CATHERINE <[email protected]> Subject: Re: [onap-discuss] [ONAP] [Intergation] [Gating]OOM Gating issue during the week end no problem I understand and again, sorry for the service interruption for the last gating https://gerrit.onap.org/r/c/oom/+/89091 1) still 8 pods in errors (OOF-related + multicloud-etcd) so unfortunately the patch does not fix the OOF issues 2) Robot healthcheck: 3 errors - AAF-SMS - OOF-Homing and OOF-SNIRO 3) Model distribution of vFW is OK 4) VNF End to End onboarding/distribution/instantiation OK for basic_vm, freeradius_nbi, quota issue for vIMS but sounds OK It would be possible to complete the existing feedback with the model distribution and the End to End tests. /Morgan Le lundi 03 juin 2019 à 15:42 +0000, FREEMAN, BRIAN D a écrit : Morgan, We get tremendous value with the script so we usually wait for it to fire and then if it looks like a system problem we proceed at risk but we really appreciate the work on keeping the environment running. We worked through it. Thanks Brian From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Sent: Monday, June 03, 2019 11:38 AM To: FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; PLATANIA, MARCO <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; LEFEVRE, CATHERINE <[email protected]<mailto:[email protected]>> Subject: [ONAP] [Intergation] [Gating]OOM Gating issue during the week end Hi for information, the Orange lab faced a RAM issue on the 31th of May at 4:35 AM UTC (140% reached on 1 compute). One of the consequences was that 1 pod was in Unknown state on Gating project (another one was more errors on our daily master). The traditional Gating sequence was then blocked because Unknown pods are not properly cleaned with the usual scripts, we need to force and set the graceful period to 0. As it was not possible to clean, all the gating attempts failed from this date to this morning when we cleaned manually the faulty pod in the project. The gating attempts on patch 89007, 89019 (2 patchsets), 88991, 89008 (2), 89061 (4), 88984 (2), 89088, 89096 (2) and 89091 (2) were not properly executed. After cleaning of the resources this morning, it was possible execute 89096 and queue the patches for new gating. I assume that some patches were merged without gating as they were blocking for some use cases (88984, 89008 (Clamp), 88991 (SO)) One patch was abandonned (89088) On 89096, we got the following results (https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/jobs/224062026<https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.com_Orange-2DOpenSource_lfn_onap_xtesting-2Donap_-2D_jobs_224062026&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=yJdyxSlfjQz1PwxldbQUKRlELiK2uwXBcc8Gn07ybBU&s=5pBfY3FrRTKiWMUzH-KkxHKA_BVUCRhGNK4J5xBsoKg&e=>): - 8 pods FAIL (OOF related pods + onap-multicloud-multicloud-k8s-etcd-0) - 4 robot healthcheck FAIL (AAF SMS health check, OOF-Homing, OOF-SNIRO, VFC vnfmgr API) - Model distribution of the vFW was FAIL (404 instead of expected 200) - End to End VNF onboarding/instantiation operations +-----------------+------------------+-----------+---------------+-----------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-----------------+------------------+-----------+---------------+-----------+ | basic_vm | functest | xtesting | 08:17 | PASS | | freeradius_nbi | functest | xtesting | 07:40 | PASS | | clearwater_ims | functest | xtesting | 10:55 | PASS | So we have a regression on the number of pods FAIL (so far only multicloud-etcd was remaining) but I can imagine that OOF issues will be solved by https://gerrit.onap.org/r/c/oom/+/89091<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.onap.org_r_c_oom_-2B_89091&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=yJdyxSlfjQz1PwxldbQUKRlELiK2uwXBcc8Gn07ybBU&s=wY3QWY1lpMfqMsWOtjEgiHchNoTTcq2Qp5lUSEQXrd0&e=> and we would be back to the situation before last week end (gating for this patch in progress) We are sorry for the inconvenience. We were not lucky as the as the first error occured on a long week-end (bank holiday on Thursday in France), closed to the release date with an abnormal number of patchsets (usually it is relatively quiete over the week end). For the future, it could make sense to distribute the gating on different sites and/or grant some external access for integration team to restore the gating project if needed. Merging without getting a feedback could be dangerous, here if we are really facing regression, it will be harder to detect the patch responsible for the regression as many were merged. /Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17388): https://lists.onap.org/g/onap-discuss/message/17388 Mute This Topic: https://lists.onap.org/mt/31911771/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
