+1 On Sun, Jun 26, 2016 at 1:40 PM, Luis Gomez <[email protected]> wrote:
> Thanks Andy and Thanh for such a big effort. > > BR/Luis > > > On Jun 26, 2016, at 12:02 PM, Andrew Grimberg < > [email protected]> wrote: > > Greetings folks, > > Just shy of 28 hours after we started the maintenance, and missing our > original window end only by 19 hours... we're now declaring the Jenkins > migration completed. > > I apologize for the really, really bad window estimate there. > > For those interested here's all the changes that happened during this > maintenance: > > * Jenkins migrated from Rackspace public cloud to private cloud. This > was the bulk of the work. We had between 2.4 and 2.5TB of data that was > syncronized between the old and new systems. This sync was running all > week and while it had finished on Thursday night, our finalized sync > with Jenkins shutdown on both ends is what took the bulk of our time. > > - Jenkins was updated to the latest LTS version (1.651.3) We had been > running on a very old LTS version as updating past it while possible on > the EL6 system it was on, was not easy with several of the plugins we > use needing newer system level services. The new Jenkins system is EL7 > > - We transitioned off of the JClouds provider plugin to the OpenStack > provider plugin for all of our current instance management > > - We reconfigured the Jenkins hosted maven settings files to a better > naming configuration so we could do away with our mapping macros in JJB > > * Nexus was updated to the latest version (2.12.0 -> 2.13.0) > > * All systems in the CI environment received the latest system updates > which we try to do on a monthly basis anyway but it was perfect timing > > * CLM was updated to the latest version (1.19 -> 1.21) > > Additional changes folks may notice: > > * Jenkins build instances will only get reused if there is sufficient > queue to support a verify or merge job hitting one as soon as a previous > job finishes as they will only idle for 1 minute now instead of 15 - 30 > minutes as we did in the public cloud > > * Instances will, generally, start much faster as only our images are > cached on the compute nodes > > * For those that have looked at the vagrant definitions we use for > managing the instance snapshots, you may notice that they're a bit > simpler. While we haven't put in the extra work to make them operational > with the standard upstream vagrant boxes available, our base images + > vagrant definitions are all completely in the open now instead of having > to base the systems on something from Rackspace that we couldn't hand to > the community. > > Finally, I want to thank Thanh for sticking with me through this > migration. It was definitely a lot longer than we had originally planned > but with all the work that he put in, along with the rest of the folks > in integration, we seem to have ironed out most of the issues before > they even showed up. > > At this point, the issues that I truly expect us to see are going to be > capacity related, so if the queues get extra long for a bit, we're > sorry. We're aware that it's a possibility with a change of this > magnitude. We're going to be watching very carefully and doing what we > can to tune things better. > > -Andy- > > On 06/26/2016 09:46 AM, Andrew Grimberg wrote: > > Status update: > > Disk management has been completed. Jenkins is online but we're still > working through the changes that need to happen after Jenkins is running > again. > > As such Jenkins is going to remain in the non-processing 'going to shut > down' mode until we have finished our changes. > > Current estimate still puts us at ~12:00 PDT before we reopen Jenkins > for proper service. > > -Andy- > > On 06/26/2016 06:04 AM, Andrew Grimberg wrote: > > Greetings folks, > > Just an update on the outage. Yes, we're still down, but we're finally > into the home stretch of disk changes before we can restart Jenkins and > then apply the needed job changes related to the Jenkins updates. > > With the present rate at which the related disk changes are happening I > am presently anticipating that we'll have it back online by 12:00 PDT > today (6 hours out). > > My apologies for the longer outage duration than originally planned! > -Andy- > > On 06/25/2016 04:55 PM, Andrew Grimberg wrote: > > Original window is about to close but we still haven't completed the > migration. From the look of things we're about 1 - 2 hours away from the > finalized disk sync finishing. Giving how long we've been down and that > we're likely to see similar lengths of time for an attempt at a later > date, we're just going to go ahead and power through. > > Sorry for the extended outage! > > -Andy- > > On 06/25/2016 07:45 AM, Andrew Grimberg wrote: > > This work will be starting in 15 minutes. > > -Andy- > > On 06/23/2016 09:31 AM, Andrew Grimberg wrote: > > What: The Linux Foundation will be performing the final migration of the > OpenDaylight Jenkins releng silo (aka production silo) to from the > Rackspace public cloud to private cloud > > When: Saturday, June 25, 2016 @ 08:00 - 17:00 PDT (15:00 - 00:00 UTC) > > Why: This is the final step in our migration from the Rackspace public > cloud to the private cloud. > > Impact: The production Jenkins system will be offline for the duration > of the migration. The size of the window is needed to account for final > data synchronization of the current jenkins silo to the new new one > which consists of ~2.3TB of data. > > Additionally, while we're doing the final disk sync we will take the > time to do needed system updates on other components of the CI > infrastructure. As such, there will be some rolling outages of Gerrit, > Nexus, Sonar, and CLM. > > We will be sending out a note to the lists and #opendaylight IRC channel > on Freenode at the beginning and end of the maintenance. > > -Andy- > > > > > > > -- > Andrew J Grimberg > Systems Administrator > Release Engineering Team Lead > The Linux Foundation > > _______________________________________________ > release mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/release > > > > _______________________________________________ > dev mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/dev > > -- Thanks Anil
_______________________________________________ Discuss mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/discuss
