no problem I understand and again, sorry for the service interruption for the last gating https://gerrit.onap.org/r/c/oom/+/89091 1) still 8 pods in errors (OOF-related + multicloud-etcd)so unfortunately the patch does not fix the OOF issues 2) Robot healthcheck: 3 errors- AAF-SMS- OOF-Homing and OOF-SNIRO 3) Model distribution of vFW is OK 4) VNF End to End onboarding/distribution/instantiation OK for basic_vm, freeradius_nbi, quota issue for vIMS but sounds OK It would be possible to complete the existing feedback with the model distribution and the End to End tests. /Morgan
Le lundi 03 juin 2019 à 15:42 +0000, FREEMAN, BRIAN D a écrit : > Morgan, > > We get tremendous value with the script so we usually wait for it to > fire and then if it looks like a system problem we proceed at risk > but we really appreciate the work on keeping the environment running. > > We worked through it. > > Thanks > > > Brian > > > > > From: [email protected] <[email protected]> > > > Sent: Monday, June 03, 2019 11:38 AM > > To: FREEMAN, BRIAN D <[email protected]>; [email protected]; > PLATANIA, MARCO <[email protected]>; [email protected] > > Cc: [email protected]; [email protected]; > [email protected]; [email protected]; LEFEVRE, CATHERINE < > [email protected]> > > Subject: [ONAP] [Intergation] [Gating]OOM Gating issue during the > week end > > > > > Hi > > > > > > for information, the Orange lab faced a RAM issue on the 31th of May > at 4:35 AM UTC (140% reached on 1 compute). > > > One of the consequences was that 1 pod was in Unknown state on Gating > project (another one was more errors on our daily master). > > > The traditional Gating sequence was then blocked because Unknown pods > are not properly cleaned with the usual scripts, we need to force and > set the graceful period to 0. > > > As it was not possible to clean, all the gating attempts failed from > this date to this morning when we cleaned manually the faulty pod in > the project. > > > > > > The gating attempts on patch 89007, 89019 (2 patchsets), 88991, 89008 > (2), 89061 (4), 88984 (2), 89088, 89096 (2) and 89091 (2) were not > properly executed. > > > > > > After cleaning of the resources this morning, it was possible execute > 89096 and queue the patches for new gating. > > > > > > I assume that some patches were merged without gating as they were > blocking for some use cases (88984, 89008 (Clamp), 88991 (SO)) > > > One patch was abandonned (89088) > > > > > > On 89096, we got the following results ( > https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/jobs/224062026 > ): > > > - 8 pods FAIL (OOF related pods + onap-multicloud-multicloud-k8s- > etcd-0) > > > - 4 robot healthcheck FAIL (AAF SMS health check, OOF-Homing, OOF- > SNIRO, VFC vnfmgr API) > > > - Model distribution of the vFW was FAIL (404 instead of expected > 200) > > > - End to End VNF onboarding/instantiation operations > > > +-----------------+------------------+-----------+---------------+--- > --------+ > > > | TEST CASE | PROJECT | TIER > | DURATION | RESULT | > > > +-----------------+------------------+-----------+---------------+--- > --------+ > > > | basic_vm | functest > | xtesting | 08:17 | PASS | > > > | freeradius_nbi | functest > | xtesting | 07:40 | PASS | > > > | clearwater_ims | functest > | xtesting | 10:55 | PASS | > > > > > > So we have a regression on the number of pods FAIL (so far only > multicloud-etcd was remaining) but I can imagine that OOF issues will > be solved by > > https://gerrit.onap.org/r/c/oom/+/89091 and we would be back to the > situation before last week end (gating for this patch in progress) > > > > > > We are sorry for the inconvenience. > > > We were not lucky as the as the first error occured on a long week- > end (bank holiday on Thursday in France), closed to the release date > with an abnormal number of patchsets (usually it is relatively quiete > over the week end). > > > > For the future, it could make sense to distribute the gating on > different sites and/or grant some external access for integration > team to restore the gating project if needed. > > > Merging without getting a feedback could be dangerous, here if we are > really facing regression, it will be harder to detect the patch > responsible for the regression as many were merged. > > > > > > /Morgan > > > > > > > > _____________________________________________________________________ > ____________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous > avez recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les > messages electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, > deforme ou falsifie. Merci. > > This message and its attachments may contain confidential or > privileged information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender > and delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have > been modified, changed or falsified. > Thank you. > > > _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17386): https://lists.onap.org/g/onap-discuss/message/17386 Mute This Topic: https://lists.onap.org/mt/31911771/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
