same here https://jenkins.opendaylight.org/releng/job/openflowplugin-distribution-check-boron/1021/console
Waiting for Jenkins to finish collecting data [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install) on project distribution-karaf: Failed to install artifact org.opendaylight.integration:distribution-karaf:tar.gz:0.5.0-SNAPSHOT: No space left on device -> [Help 1] ________________________________ From: [email protected] <[email protected]> on behalf of Anil Vishnoi <[email protected]> Sent: Monday, June 27, 2016 1:42 AM To: Andrew Grimberg Cc: OpenDaylight Dev; OpenDaylight Discuss; IT Infrastructure Alerts; [email protected]; OpenDaylight Infrastructure Subject: Re: [release] OpenDaylight Jenkins releng (production) maintenance 2016-06-25 @08:00 - 17:00 PDT even the verify job is failing for the same reason https://jenkins.opendaylight.org/releng/job/neutron-verify-boron/310/jdk=openjdk8,nodes=dynamic_verify/console Caused by: hudson.plugins.git.GitException: Command "git init /w/workspace/neutron-verify-boron/jdk/openjdk8/nodes/dynamic_verify" returned status code 128: stdout: stderr: fatal: cannot copy '/usr/share/git-core/templates/hooks/pre-applypatch.sample' to '/w/workspace/neutron-verify-boron/jdk/openjdk8/nodes/dynamic_verify/.git/hooks/pre-applypatch.sample': No space left on device On Sun, Jun 26, 2016 at 4:39 PM, Anil Vishnoi <[email protected]<mailto:[email protected]>> wrote: Hi Andy/Thanh, For one of my patch distribution job is failing because of disk space issue. https://jenkins.opendaylight.org/releng/job/neutron-distribution-check-boron/248/console [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install) on project distribution-karaf: Failed to install artifact org.opendaylight.integration:distribution-karaf:zip:0.5.0-SNAPSHOT: No space left on device -> [Help 1] On Sun, Jun 26, 2016 at 12:02 PM, Andrew Grimberg <[email protected]<mailto:[email protected]>> wrote: Greetings folks, Just shy of 28 hours after we started the maintenance, and missing our original window end only by 19 hours... we're now declaring the Jenkins migration completed. I apologize for the really, really bad window estimate there. For those interested here's all the changes that happened during this maintenance: * Jenkins migrated from Rackspace public cloud to private cloud. This was the bulk of the work. We had between 2.4 and 2.5TB of data that was syncronized between the old and new systems. This sync was running all week and while it had finished on Thursday night, our finalized sync with Jenkins shutdown on both ends is what took the bulk of our time. - Jenkins was updated to the latest LTS version (1.651.3) We had been running on a very old LTS version as updating past it while possible on the EL6 system it was on, was not easy with several of the plugins we use needing newer system level services. The new Jenkins system is EL7 - We transitioned off of the JClouds provider plugin to the OpenStack provider plugin for all of our current instance management - We reconfigured the Jenkins hosted maven settings files to a better naming configuration so we could do away with our mapping macros in JJB * Nexus was updated to the latest version (2.12.0 -> 2.13.0) * All systems in the CI environment received the latest system updates which we try to do on a monthly basis anyway but it was perfect timing * CLM was updated to the latest version (1.19 -> 1.21) Additional changes folks may notice: * Jenkins build instances will only get reused if there is sufficient queue to support a verify or merge job hitting one as soon as a previous job finishes as they will only idle for 1 minute now instead of 15 - 30 minutes as we did in the public cloud * Instances will, generally, start much faster as only our images are cached on the compute nodes * For those that have looked at the vagrant definitions we use for managing the instance snapshots, you may notice that they're a bit simpler. While we haven't put in the extra work to make them operational with the standard upstream vagrant boxes available, our base images + vagrant definitions are all completely in the open now instead of having to base the systems on something from Rackspace that we couldn't hand to the community. Finally, I want to thank Thanh for sticking with me through this migration. It was definitely a lot longer than we had originally planned but with all the work that he put in, along with the rest of the folks in integration, we seem to have ironed out most of the issues before they even showed up. At this point, the issues that I truly expect us to see are going to be capacity related, so if the queues get extra long for a bit, we're sorry. We're aware that it's a possibility with a change of this magnitude. We're going to be watching very carefully and doing what we can to tune things better. -Andy- On 06/26/2016 09:46 AM, Andrew Grimberg wrote: > Status update: > > Disk management has been completed. Jenkins is online but we're still > working through the changes that need to happen after Jenkins is running > again. > > As such Jenkins is going to remain in the non-processing 'going to shut > down' mode until we have finished our changes. > > Current estimate still puts us at ~12:00 PDT before we reopen Jenkins > for proper service. > > -Andy- > > On 06/26/2016 06:04 AM, Andrew Grimberg wrote: >> Greetings folks, >> >> Just an update on the outage. Yes, we're still down, but we're finally >> into the home stretch of disk changes before we can restart Jenkins and >> then apply the needed job changes related to the Jenkins updates. >> >> With the present rate at which the related disk changes are happening I >> am presently anticipating that we'll have it back online by 12:00 PDT >> today (6 hours out). >> >> My apologies for the longer outage duration than originally planned! >> -Andy- >> >> On 06/25/2016 04:55 PM, Andrew Grimberg wrote: >>> Original window is about to close but we still haven't completed the >>> migration. From the look of things we're about 1 - 2 hours away from the >>> finalized disk sync finishing. Giving how long we've been down and that >>> we're likely to see similar lengths of time for an attempt at a later >>> date, we're just going to go ahead and power through. >>> >>> Sorry for the extended outage! >>> >>> -Andy- >>> >>> On 06/25/2016 07:45 AM, Andrew Grimberg wrote: >>>> This work will be starting in 15 minutes. >>>> >>>> -Andy- >>>> >>>> On 06/23/2016 09:31 AM, Andrew Grimberg wrote: >>>>> What: The Linux Foundation will be performing the final migration of the >>>>> OpenDaylight Jenkins releng silo (aka production silo) to from the >>>>> Rackspace public cloud to private cloud >>>>> >>>>> When: Saturday, June 25, 2016 @ 08:00 - 17:00 PDT (15:00 - 00:00 UTC) >>>>> >>>>> Why: This is the final step in our migration from the Rackspace public >>>>> cloud to the private cloud. >>>>> >>>>> Impact: The production Jenkins system will be offline for the duration >>>>> of the migration. The size of the window is needed to account for final >>>>> data synchronization of the current jenkins silo to the new new one >>>>> which consists of ~2.3TB of data. >>>>> >>>>> Additionally, while we're doing the final disk sync we will take the >>>>> time to do needed system updates on other components of the CI >>>>> infrastructure. As such, there will be some rolling outages of Gerrit, >>>>> Nexus, Sonar, and CLM. >>>>> >>>>> We will be sending out a note to the lists and #opendaylight IRC channel >>>>> on Freenode at the beginning and end of the maintenance. >>>>> >>>>> -Andy- >>>>> >>>> >>> >> > -- Andrew J Grimberg Systems Administrator Release Engineering Team Lead The Linux Foundation _______________________________________________ release mailing list [email protected]<mailto:[email protected]> https://lists.opendaylight.org/mailman/listinfo/release -- Thanks Anil -- Thanks Anil
_______________________________________________ Discuss mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/discuss
