Hi for information, the Orange lab faced a RAM issue on the 31th of May at 4:35 AM UTC (140% reached on 1 compute). One of the consequences was that 1 pod was in Unknown state on Gating project (another one was more errors on our daily master). The traditional Gating sequence was then blocked because Unknown pods are not properly cleaned with the usual scripts, we need to force and set the graceful period to 0. As it was not possible to clean, all the gating attempts failed from this date to this morning when we cleaned manually the faulty pod in the project.
The gating attempts on patch 89007, 89019 (2 patchsets), 88991, 89008 (2), 89061 (4), 88984 (2), 89088, 89096 (2) and 89091 (2) were not properly executed. After cleaning of the resources this morning, it was possible execute 89096 and queue the patches for new gating. I assume that some patches were merged without gating as they were blocking for some use cases (88984, 89008 (Clamp), 88991 (SO)) One patch was abandonned (89088) On 89096, we got the following results ( https://gitlab.com/Orange-OpenSource/lfn/onap/xtesting-onap/-/jobs/224062026 ): - 8 pods FAIL (OOF related pods + onap-multicloud-multicloud-k8s-etcd- 0) - 4 robot healthcheck FAIL (AAF SMS health check, OOF-Homing, OOF- SNIRO, VFC vnfmgr API) - Model distribution of the vFW was FAIL (404 instead of expected 200) - End to End VNF onboarding/instantiation operations +-----------------+------------------+-----------+---------------+----- ------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-----------------+------------------+-----------+---------------+----- ------+ | basic_vm | functest | xtesting | 08:17 | PASS | | freeradius_nbi | functest | xtesting | 07:40 | PASS | | clearwater_ims | functest | xtesting | 10:55 | PASS | So we have a regression on the number of pods FAIL (so far only multicloud-etcd was remaining) but I can imagine that OOF issues will be solved by https://gerrit.onap.org/r/c/oom/+/89091 and we would be back to the situation before last week end (gating for this patch in progress) We are sorry for the inconvenience. We were not lucky as the as the first error occured on a long week-end (bank holiday on Thursday in France), closed to the release date with an abnormal number of patchsets (usually it is relatively quiete over the week end). For the future, it could make sense to distribute the gating on different sites and/or grant some external access for integration team to restore the gating project if needed. Merging without getting a feedback could be dangerous, here if we are really facing regression, it will be harder to detect the patch responsible for the regression as many were merged. /Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17384): https://lists.onap.org/g/onap-discuss/message/17384 Mute This Topic: https://lists.onap.org/mt/31911771/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
