Couple of comments regarding distribution test duration: - I saw the netconf scale test can take from 6 to 10 hours so I filled this patch to reduce that time, otherwise we can move it to the weekend like the longevity tests:
https://git.opendaylight.org/gerrit/c/releng/builder/+/97011 <https://git.opendaylight.org/gerrit/c/releng/builder/+/97011> - AR Aluminium is still running once a day, should we set this to weekly? AFAIR we did that for the 2nd stable branch in the past. https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-aluminium-mvn35-openjdk11/ <https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-aluminium-mvn35-openjdk11/> BR/Luis > On Jul 21, 2021, at 3:46 PM, Robert Varga <[email protected]> wrote: > > So this again is interplay between CSIT, Vexxhost, openstack and us. > > There were wide-scale Jenkins problems during past 24 hours, as for example: > > 1. > https://jenkins.opendaylight.org/releng/job/integration-distribution-test-silicon/390/ > spent 12 hours waiting in the queue. This job normally executes within > 90 minutes. > > 2. > https://jenkins.opendaylight.org/releng/view/distribution/job/integration-distribution-test-aluminium/507/ > has 2 new failures in three-node scenarios > > 3. > https://jenkins.opendaylight.org/releng/view/distribution/job/distribution-merge-managed-aluminium/4928/ > took 13 hours to build. > It usually completes in 20 minutes. > > https://jenkins.opendaylight.org/releng/view/distribution/job/distribution-merge-managed-aluminium/4928/console > reveals this: > >> 01:03:35 -----END_OF_BUILD----- > > This is after end of build, i.e. post-build actions which the ODL > community has no way of affecting. > >> 01:03:35 [JaCoCo plugin] Loading packages.. >> 12:21:28 [JaCoCo plugin] Done. > > i.e. it took 11 hours and 18 minutes... > >> 12:21:28 [JaCoCo plugin] Loading packages.. >> 12:26:24 [JaCoCo plugin] Done. > > ... then 5 minutes ... > >> 12:26:24 [JaCoCo plugin] Loading packages.. >> 13:43:35 [JaCoCo plugin] Done. > > ... and some 76 minutes ... > >> 13:43:35 [JaCoCo plugin] Loading packages.. >> 14:25:10 [JaCoCo plugin] Done. > > ... and some 42 minutes ... > >> 14:25:10 [JaCoCo plugin] Loading packages.. >> 14:25:20 [JaCoCo plugin] Done. >> 14:25:20 [PostBuildScript] - [ERROR] An error occured during post-build >> processing. >> 14:25:32 org.jenkinsci.plugins.postbuildscript.PostBuildScriptException: >> java.lang.InterruptedException > > and this is me, taking that job out back. It probably would still be > running, holding back retriggers (of which 7 have since passed > successfully). > > While all of this was going on, https://status.linuxfoundation.org/ > shows OpenDaylight infra to be happy as a clam the last 24 hours. > > Something is seriously not adding up here and it is Not Fun[*]. > > Regards, > Robert > > [*] All of this delayed Phosphorus CSIT results by ~10 hours, i.e. no > new data during morning CET. > > > On 21/07/2021 22:22, Project Services wrote: >> —-—-—-— >> —-—-—-— >> Reply above this line. >> >> Hello *Robert Varga*, >> >> *Eric Ball* has added a comment on your request *IT-22529* >> <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>: >> >> _The resource usage is currently low, so I was able to successfully >> re-run the ovsdb job. I'm re-running the openflowplugin job now. >> >> I'll also submit a request to Vexxhost to expand or remove the >> memory cap. We have a limit to the total number of instances active >> at once, which should keep our resource usage from hitting any of >> the usage caps. But it seems that the memory cap doesn't take into >> account the ram used by stacks dynamically spun up by the jobs, so a >> large number of those jobs running concurrently can still have this >> issue._ >> >> You may reply to their comment *here >> <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>* >> >> ------------------------------------------------------------------------ >> >> Hello *Robert Varga*, >> >> *System Administrator* changed the status of your request *IT-22529 >> <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email>* >> to *Waiting for customer*. >> >> View request >> <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529?sda_source=notification-email> >> · Turn off this request's notifications >> <https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-22529/unsubscribe?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJxc2giOiI0Nzc2OWVjZGY5OTNhYzM4ZjczYTNlMWZhNTY0YWViOGZmODM0ZjYzZWYyZjEzMjUzMTQ1NjU0OTg5ZGYwMTMyIiwiaXNzIjoic2VydmljZWRlc2stand0LXRva2VuLWlzc3VlciIsImNvbnRleHQiOnsidXNlciI6InJvdmFyZ2EiLCJpc3N1ZSI6IklULTIyNTI5In0sImV4cCI6MTYyOTMxODEwNCwiaWF0IjoxNjI2ODk4OTA0fQ.Dw61-8vZTZJFaz5psnA6RauRWMALf-ctSJIQun61XwY> >> >> >> This is shared with ODL and Robert Varga. >> >> Help Center, powered by Jira Service Desk >> <https://www.atlassian.com/software/jira/service-desk/powered-by?utm_medium=email&utm_source=service-desk_email-notification_server&utm_campaign=service-desk_email-notification_server>, >> sent you this message. >> > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1218): https://lists.opendaylight.org/g/infrastructure/message/1218 Mute This Topic: https://lists.opendaylight.org/mt/84368085/21656 Group Owner: [email protected] Unsubscribe: https://lists.opendaylight.org/g/infrastructure/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
