Hi Andy/Thanh, For one of my patch distribution job is failing because of disk space issue.
https://jenkins.opendaylight.org/releng/job/neutron-distribution-check-boron/248/console [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install) on project distribution-karaf: Failed to install artifact org.opendaylight.integration:distribution-karaf:zip:0.5.0-SNAPSHOT: No space left on device -> [Help 1] On Sun, Jun 26, 2016 at 12:02 PM, Andrew Grimberg < [email protected]> wrote: > Greetings folks, > > Just shy of 28 hours after we started the maintenance, and missing our > original window end only by 19 hours... we're now declaring the Jenkins > migration completed. > > I apologize for the really, really bad window estimate there. > > For those interested here's all the changes that happened during this > maintenance: > > * Jenkins migrated from Rackspace public cloud to private cloud. This > was the bulk of the work. We had between 2.4 and 2.5TB of data that was > syncronized between the old and new systems. This sync was running all > week and while it had finished on Thursday night, our finalized sync > with Jenkins shutdown on both ends is what took the bulk of our time. > > - Jenkins was updated to the latest LTS version (1.651.3) We had been > running on a very old LTS version as updating past it while possible on > the EL6 system it was on, was not easy with several of the plugins we > use needing newer system level services. The new Jenkins system is EL7 > > - We transitioned off of the JClouds provider plugin to the OpenStack > provider plugin for all of our current instance management > > - We reconfigured the Jenkins hosted maven settings files to a better > naming configuration so we could do away with our mapping macros in JJB > > * Nexus was updated to the latest version (2.12.0 -> 2.13.0) > > * All systems in the CI environment received the latest system updates > which we try to do on a monthly basis anyway but it was perfect timing > > * CLM was updated to the latest version (1.19 -> 1.21) > > Additional changes folks may notice: > > * Jenkins build instances will only get reused if there is sufficient > queue to support a verify or merge job hitting one as soon as a previous > job finishes as they will only idle for 1 minute now instead of 15 - 30 > minutes as we did in the public cloud > > * Instances will, generally, start much faster as only our images are > cached on the compute nodes > > * For those that have looked at the vagrant definitions we use for > managing the instance snapshots, you may notice that they're a bit > simpler. While we haven't put in the extra work to make them operational > with the standard upstream vagrant boxes available, our base images + > vagrant definitions are all completely in the open now instead of having > to base the systems on something from Rackspace that we couldn't hand to > the community. > > Finally, I want to thank Thanh for sticking with me through this > migration. It was definitely a lot longer than we had originally planned > but with all the work that he put in, along with the rest of the folks > in integration, we seem to have ironed out most of the issues before > they even showed up. > > At this point, the issues that I truly expect us to see are going to be > capacity related, so if the queues get extra long for a bit, we're > sorry. We're aware that it's a possibility with a change of this > magnitude. We're going to be watching very carefully and doing what we > can to tune things better. > > -Andy- > > On 06/26/2016 09:46 AM, Andrew Grimberg wrote: > > Status update: > > > > Disk management has been completed. Jenkins is online but we're still > > working through the changes that need to happen after Jenkins is running > > again. > > > > As such Jenkins is going to remain in the non-processing 'going to shut > > down' mode until we have finished our changes. > > > > Current estimate still puts us at ~12:00 PDT before we reopen Jenkins > > for proper service. > > > > -Andy- > > > > On 06/26/2016 06:04 AM, Andrew Grimberg wrote: > >> Greetings folks, > >> > >> Just an update on the outage. Yes, we're still down, but we're finally > >> into the home stretch of disk changes before we can restart Jenkins and > >> then apply the needed job changes related to the Jenkins updates. > >> > >> With the present rate at which the related disk changes are happening I > >> am presently anticipating that we'll have it back online by 12:00 PDT > >> today (6 hours out). > >> > >> My apologies for the longer outage duration than originally planned! > >> -Andy- > >> > >> On 06/25/2016 04:55 PM, Andrew Grimberg wrote: > >>> Original window is about to close but we still haven't completed the > >>> migration. From the look of things we're about 1 - 2 hours away from > the > >>> finalized disk sync finishing. Giving how long we've been down and that > >>> we're likely to see similar lengths of time for an attempt at a later > >>> date, we're just going to go ahead and power through. > >>> > >>> Sorry for the extended outage! > >>> > >>> -Andy- > >>> > >>> On 06/25/2016 07:45 AM, Andrew Grimberg wrote: > >>>> This work will be starting in 15 minutes. > >>>> > >>>> -Andy- > >>>> > >>>> On 06/23/2016 09:31 AM, Andrew Grimberg wrote: > >>>>> What: The Linux Foundation will be performing the final migration of > the > >>>>> OpenDaylight Jenkins releng silo (aka production silo) to from the > >>>>> Rackspace public cloud to private cloud > >>>>> > >>>>> When: Saturday, June 25, 2016 @ 08:00 - 17:00 PDT (15:00 - 00:00 UTC) > >>>>> > >>>>> Why: This is the final step in our migration from the Rackspace > public > >>>>> cloud to the private cloud. > >>>>> > >>>>> Impact: The production Jenkins system will be offline for the > duration > >>>>> of the migration. The size of the window is needed to account for > final > >>>>> data synchronization of the current jenkins silo to the new new one > >>>>> which consists of ~2.3TB of data. > >>>>> > >>>>> Additionally, while we're doing the final disk sync we will take the > >>>>> time to do needed system updates on other components of the CI > >>>>> infrastructure. As such, there will be some rolling outages of > Gerrit, > >>>>> Nexus, Sonar, and CLM. > >>>>> > >>>>> We will be sending out a note to the lists and #opendaylight IRC > channel > >>>>> on Freenode at the beginning and end of the maintenance. > >>>>> > >>>>> -Andy- > >>>>> > >>>> > >>> > >> > > > > -- > Andrew J Grimberg > Systems Administrator > Release Engineering Team Lead > The Linux Foundation > > > _______________________________________________ > release mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/release > > -- Thanks Anil
_______________________________________________ Discuss mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/discuss
