even the verify job is failing for the same reason https://jenkins.opendaylight.org/releng/job/neutron-verify-boron/310/jdk=openjdk8,nodes=dynamic_verify/console
Caused by: hudson.plugins.git.GitException: Command "git init /w/workspace/neutron-verify-boron/jdk/openjdk8/nodes/dynamic_verify" returned status code 128: stdout: stderr: fatal: cannot copy '/usr/share/git-core/templates/hooks/pre-applypatch.sample' to '/w/workspace/neutron-verify-boron/jdk/openjdk8/nodes/dynamic_verify/.git/hooks/pre-applypatch.sample': No space left on device On Sun, Jun 26, 2016 at 4:39 PM, Anil Vishnoi <[email protected]> wrote: > Hi Andy/Thanh, > > For one of my patch distribution job is failing because of disk space > issue. > > > https://jenkins.opendaylight.org/releng/job/neutron-distribution-check-boron/248/console > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install) > on project distribution-karaf: Failed to install artifact > org.opendaylight.integration:distribution-karaf:zip:0.5.0-SNAPSHOT: No space > left on device -> [Help 1] > > > On Sun, Jun 26, 2016 at 12:02 PM, Andrew Grimberg < > [email protected]> wrote: > >> Greetings folks, >> >> Just shy of 28 hours after we started the maintenance, and missing our >> original window end only by 19 hours... we're now declaring the Jenkins >> migration completed. >> >> I apologize for the really, really bad window estimate there. >> >> For those interested here's all the changes that happened during this >> maintenance: >> >> * Jenkins migrated from Rackspace public cloud to private cloud. This >> was the bulk of the work. We had between 2.4 and 2.5TB of data that was >> syncronized between the old and new systems. This sync was running all >> week and while it had finished on Thursday night, our finalized sync >> with Jenkins shutdown on both ends is what took the bulk of our time. >> >> - Jenkins was updated to the latest LTS version (1.651.3) We had been >> running on a very old LTS version as updating past it while possible on >> the EL6 system it was on, was not easy with several of the plugins we >> use needing newer system level services. The new Jenkins system is EL7 >> >> - We transitioned off of the JClouds provider plugin to the OpenStack >> provider plugin for all of our current instance management >> >> - We reconfigured the Jenkins hosted maven settings files to a better >> naming configuration so we could do away with our mapping macros in JJB >> >> * Nexus was updated to the latest version (2.12.0 -> 2.13.0) >> >> * All systems in the CI environment received the latest system updates >> which we try to do on a monthly basis anyway but it was perfect timing >> >> * CLM was updated to the latest version (1.19 -> 1.21) >> >> Additional changes folks may notice: >> >> * Jenkins build instances will only get reused if there is sufficient >> queue to support a verify or merge job hitting one as soon as a previous >> job finishes as they will only idle for 1 minute now instead of 15 - 30 >> minutes as we did in the public cloud >> >> * Instances will, generally, start much faster as only our images are >> cached on the compute nodes >> >> * For those that have looked at the vagrant definitions we use for >> managing the instance snapshots, you may notice that they're a bit >> simpler. While we haven't put in the extra work to make them operational >> with the standard upstream vagrant boxes available, our base images + >> vagrant definitions are all completely in the open now instead of having >> to base the systems on something from Rackspace that we couldn't hand to >> the community. >> >> Finally, I want to thank Thanh for sticking with me through this >> migration. It was definitely a lot longer than we had originally planned >> but with all the work that he put in, along with the rest of the folks >> in integration, we seem to have ironed out most of the issues before >> they even showed up. >> >> At this point, the issues that I truly expect us to see are going to be >> capacity related, so if the queues get extra long for a bit, we're >> sorry. We're aware that it's a possibility with a change of this >> magnitude. We're going to be watching very carefully and doing what we >> can to tune things better. >> >> -Andy- >> >> On 06/26/2016 09:46 AM, Andrew Grimberg wrote: >> > Status update: >> > >> > Disk management has been completed. Jenkins is online but we're still >> > working through the changes that need to happen after Jenkins is running >> > again. >> > >> > As such Jenkins is going to remain in the non-processing 'going to shut >> > down' mode until we have finished our changes. >> > >> > Current estimate still puts us at ~12:00 PDT before we reopen Jenkins >> > for proper service. >> > >> > -Andy- >> > >> > On 06/26/2016 06:04 AM, Andrew Grimberg wrote: >> >> Greetings folks, >> >> >> >> Just an update on the outage. Yes, we're still down, but we're finally >> >> into the home stretch of disk changes before we can restart Jenkins and >> >> then apply the needed job changes related to the Jenkins updates. >> >> >> >> With the present rate at which the related disk changes are happening I >> >> am presently anticipating that we'll have it back online by 12:00 PDT >> >> today (6 hours out). >> >> >> >> My apologies for the longer outage duration than originally planned! >> >> -Andy- >> >> >> >> On 06/25/2016 04:55 PM, Andrew Grimberg wrote: >> >>> Original window is about to close but we still haven't completed the >> >>> migration. From the look of things we're about 1 - 2 hours away from >> the >> >>> finalized disk sync finishing. Giving how long we've been down and >> that >> >>> we're likely to see similar lengths of time for an attempt at a later >> >>> date, we're just going to go ahead and power through. >> >>> >> >>> Sorry for the extended outage! >> >>> >> >>> -Andy- >> >>> >> >>> On 06/25/2016 07:45 AM, Andrew Grimberg wrote: >> >>>> This work will be starting in 15 minutes. >> >>>> >> >>>> -Andy- >> >>>> >> >>>> On 06/23/2016 09:31 AM, Andrew Grimberg wrote: >> >>>>> What: The Linux Foundation will be performing the final migration >> of the >> >>>>> OpenDaylight Jenkins releng silo (aka production silo) to from the >> >>>>> Rackspace public cloud to private cloud >> >>>>> >> >>>>> When: Saturday, June 25, 2016 @ 08:00 - 17:00 PDT (15:00 - 00:00 >> UTC) >> >>>>> >> >>>>> Why: This is the final step in our migration from the Rackspace >> public >> >>>>> cloud to the private cloud. >> >>>>> >> >>>>> Impact: The production Jenkins system will be offline for the >> duration >> >>>>> of the migration. The size of the window is needed to account for >> final >> >>>>> data synchronization of the current jenkins silo to the new new one >> >>>>> which consists of ~2.3TB of data. >> >>>>> >> >>>>> Additionally, while we're doing the final disk sync we will take the >> >>>>> time to do needed system updates on other components of the CI >> >>>>> infrastructure. As such, there will be some rolling outages of >> Gerrit, >> >>>>> Nexus, Sonar, and CLM. >> >>>>> >> >>>>> We will be sending out a note to the lists and #opendaylight IRC >> channel >> >>>>> on Freenode at the beginning and end of the maintenance. >> >>>>> >> >>>>> -Andy- >> >>>>> >> >>>> >> >>> >> >> >> > >> >> -- >> Andrew J Grimberg >> Systems Administrator >> Release Engineering Team Lead >> The Linux Foundation >> >> >> _______________________________________________ >> release mailing list >> [email protected] >> https://lists.opendaylight.org/mailman/listinfo/release >> >> > > > -- > Thanks > Anil > -- Thanks Anil
_______________________________________________ Discuss mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/discuss
