So this again is interplay between CSIT, Vexxhost, openstack and us. There were wide-scale Jenkins problems during past 24 hours, as for example:
1. https://jenkins.opendaylight.org/releng/job/integration-distribution-test-silicon/390/ spent 12 hours waiting in the queue. This job normally executes within 90 minutes. 2. https://jenkins.opendaylight.org/releng/view/distribution/job/integration-distribution-test-aluminium/507/ has 2 new failures in three-node scenarios 3. https://jenkins.opendaylight.org/releng/view/distribution/job/distribution-merge-managed-aluminium/4928/ took 13 hours to build. It usually completes in 20 minutes. https://jenkins.opendaylight.org/releng/view/distribution/job/distribution-merge-managed-aluminium/4928/console reveals this: > 01:03:35 -----END_OF_BUILD----- This is after end of build, i.e. post-build actions which the ODL community has no way of affecting. > 01:03:35 [JaCoCo plugin] Loading packages.. > 12:21:28 [JaCoCo plugin] Done. i.e. it took 11 hours and 18 minutes... > 12:21:28 [JaCoCo plugin] Loading packages.. > 12:26:24 [JaCoCo plugin] Done. ... then 5 minutes ... > 12:26:24 [JaCoCo plugin] Loading packages.. > 13:43:35 [JaCoCo plugin] Done. ... and some 76 minutes ... > 13:43:35 [JaCoCo plugin] Loading packages.. > 14:25:10 [JaCoCo plugin] Done. ... and some 42 minutes ... > 14:25:10 [JaCoCo plugin] Loading packages.. > 14:25:20 [JaCoCo plugin] Done. > 14:25:20 [PostBuildScript] - [ERROR] An error occured during post-build > processing. > 14:25:32 org.jenkinsci.plugins.postbuildscript.PostBuildScriptException: > java.lang.InterruptedException and this is me, taking that job out back. It probably would still be running, holding back retriggers (of which 7 have since passed successfully). While all of this was going on, https://status.linuxfoundation.org/ shows OpenDaylight infra to be happy as a clam the last 24 hours. Something is seriously not adding up here and it is Not Fun[*]. Regards, Robert [*] All of this delayed Phosphorus CSIT results by ~10 hours, i.e. no new data during morning CET. On 21/07/2021 22:22, Project Services wrote: > —-—-—-— > —-—-—-— > Reply above this line. > > Hello *Robert Varga*, > > *Eric Ball* has added a comment on your request *IT-22529* > <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>: > > _The resource usage is currently low, so I was able to successfully > re-run the ovsdb job. I'm re-running the openflowplugin job now. > > I'll also submit a request to Vexxhost to expand or remove the > memory cap. We have a limit to the total number of instances active > at once, which should keep our resource usage from hitting any of > the usage caps. But it seems that the memory cap doesn't take into > account the ram used by stacks dynamically spun up by the jobs, so a > large number of those jobs running concurrently can still have this > issue._ > > You may reply to their comment *here > <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>* > > ------------------------------------------------------------------------ > > Hello *Robert Varga*, > > *System Administrator* changed the status of your request *IT-22529 > <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>* > to *Waiting for customer*. > > View request > <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email> > · Turn off this request's notifications > <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529/unsubscribe?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJxc2giOiI0Nzc2OWVjZGY5OTNhYzM4ZjczYTNlMWZhNTY0YWViOGZmODM0ZjYzZWYyZjEzMjUzMTQ1NjU0OTg5ZGYwMTMyIiwiaXNzIjoic2VydmljZWRlc2stand0LXRva2VuLWlzc3VlciIsImNvbnRleHQiOnsidXNlciI6InJvdmFyZ2EiLCJpc3N1ZSI6IklULTIyNTI5In0sImV4cCI6MTYyOTMxODEwNCwiaWF0IjoxNjI2ODk4OTA0fQ.Dw61-8vZTZJFaz5psnA6RauRWMALf-ctSJIQun61XwY> > > > This is shared with ODL and Robert Varga. > > Help Center, powered by Jira Service Desk > <https://www.atlassian.com/software/jira/service-desk/powered-by?utm_medium=email&utm_source=service-desk_email-notification_server&utm_campaign=service-desk_email-notification_server>, > sent you this message. >
OpenPGP_signature
Description: OpenPGP digital signature
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1217): https://lists.opendaylight.org/g/infrastructure/message/1217 Mute This Topic: https://lists.opendaylight.org/mt/84368085/21656 Group Owner: [email protected] Unsubscribe: https://lists.opendaylight.org/g/infrastructure/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
