Please let us know if you continue to see the space issue on new job runs. At this point it _should_ be resolved with the patch we merged to enable archive-builds macro which includes a patch to cleanup the workspace at the end of builds:
https://git.opendaylight.org/gerrit/40887/ Regards, Thanh On 27 June 2016 at 12:21, Thanh Ha <[email protected]> wrote: > Hi Folks, > > Just want to say that we think we have a fix for this and are currently > testing it. We'll deploy it asap once we run a few tests to make sure it > works. > > We think the issue is that we're spinning up VMs with less disk space than > we did on the public cloud because jobs don't need 100GB of disk space. We > think this issue appears when a VM is reused a few times and since jobs > don't cleanup the workspace at the end of the build when a new job takes > over it has less available space. > > Our working patch will deploy the Workspace cleanup plugin and force the > workspace to be cleared up at the end of each build. Will update again once > we have this deployed. > > Regards, > Thanh > > On 27 June 2016 at 06:46, Andrej Leitner -X (anleitne - PANTHEON > TECHNOLOGIES at Cisco) <[email protected]> wrote: > >> same here >> >> >> https://jenkins.opendaylight.org/releng/job/openflowplugin-distribution-check-boron/1021/console >> >> Waiting for Jenkins to finish collecting data[ERROR] Failed to execute goal >> org.apache.maven.plugins:maven-install-plugin:2.5.2:install >> (default-install) on project distribution-karaf: >> Failed to install artifact >> org.opendaylight.integration:distribution-karaf:tar.gz:0.5.0-SNAPSHOT: No >> space left on device -> [Help 1] >> >> >> ------------------------------ >> *From:* [email protected] < >> [email protected]> on behalf of Anil Vishnoi < >> [email protected]> >> *Sent:* Monday, June 27, 2016 1:42 AM >> *To:* Andrew Grimberg >> *Cc:* OpenDaylight Dev; OpenDaylight Discuss; IT Infrastructure Alerts; >> [email protected]; OpenDaylight Infrastructure >> *Subject:* Re: [release] OpenDaylight Jenkins releng (production) >> maintenance 2016-06-25 @08:00 - 17:00 PDT >> >> even the verify job is failing for the same reason >> >> >> https://jenkins.opendaylight.org/releng/job/neutron-verify-boron/310/jdk=openjdk8,nodes=dynamic_verify/console >> >> Caused by: hudson.plugins.git.GitException: Command "git init >> /w/workspace/neutron-verify-boron/jdk/openjdk8/nodes/dynamic_verify" >> returned status code 128: >> stdout: >> stderr: fatal: cannot copy >> '/usr/share/git-core/templates/hooks/pre-applypatch.sample' to >> '/w/workspace/neutron-verify-boron/jdk/openjdk8/nodes/dynamic_verify/.git/hooks/pre-applypatch.sample': >> No space left on device >> >> >> On Sun, Jun 26, 2016 at 4:39 PM, Anil Vishnoi <[email protected]> >> wrote: >> >>> Hi Andy/Thanh, >>> >>> For one of my patch distribution job is failing because of disk space >>> issue. >>> >>> >>> https://jenkins.opendaylight.org/releng/job/neutron-distribution-check-boron/248/console >>> >>> [ERROR] Failed to execute goal >>> org.apache.maven.plugins:maven-install-plugin:2.5.2:install >>> (default-install) on project distribution-karaf: Failed to install artifact >>> org.opendaylight.integration:distribution-karaf:zip:0.5.0-SNAPSHOT: No >>> space left on device -> [Help 1] >>> >>> >>> On Sun, Jun 26, 2016 at 12:02 PM, Andrew Grimberg < >>> [email protected]> wrote: >>> >>>> Greetings folks, >>>> >>>> Just shy of 28 hours after we started the maintenance, and missing our >>>> original window end only by 19 hours... we're now declaring the Jenkins >>>> migration completed. >>>> >>>> I apologize for the really, really bad window estimate there. >>>> >>>> For those interested here's all the changes that happened during this >>>> maintenance: >>>> >>>> * Jenkins migrated from Rackspace public cloud to private cloud. This >>>> was the bulk of the work. We had between 2.4 and 2.5TB of data that was >>>> syncronized between the old and new systems. This sync was running all >>>> week and while it had finished on Thursday night, our finalized sync >>>> with Jenkins shutdown on both ends is what took the bulk of our time. >>>> >>>> - Jenkins was updated to the latest LTS version (1.651.3) We had been >>>> running on a very old LTS version as updating past it while possible on >>>> the EL6 system it was on, was not easy with several of the plugins we >>>> use needing newer system level services. The new Jenkins system is EL7 >>>> >>>> - We transitioned off of the JClouds provider plugin to the OpenStack >>>> provider plugin for all of our current instance management >>>> >>>> - We reconfigured the Jenkins hosted maven settings files to a better >>>> naming configuration so we could do away with our mapping macros in JJB >>>> >>>> * Nexus was updated to the latest version (2.12.0 -> 2.13.0) >>>> >>>> * All systems in the CI environment received the latest system updates >>>> which we try to do on a monthly basis anyway but it was perfect timing >>>> >>>> * CLM was updated to the latest version (1.19 -> 1.21) >>>> >>>> Additional changes folks may notice: >>>> >>>> * Jenkins build instances will only get reused if there is sufficient >>>> queue to support a verify or merge job hitting one as soon as a previous >>>> job finishes as they will only idle for 1 minute now instead of 15 - 30 >>>> minutes as we did in the public cloud >>>> >>>> * Instances will, generally, start much faster as only our images are >>>> cached on the compute nodes >>>> >>>> * For those that have looked at the vagrant definitions we use for >>>> managing the instance snapshots, you may notice that they're a bit >>>> simpler. While we haven't put in the extra work to make them operational >>>> with the standard upstream vagrant boxes available, our base images + >>>> vagrant definitions are all completely in the open now instead of having >>>> to base the systems on something from Rackspace that we couldn't hand to >>>> the community. >>>> >>>> Finally, I want to thank Thanh for sticking with me through this >>>> migration. It was definitely a lot longer than we had originally planned >>>> but with all the work that he put in, along with the rest of the folks >>>> in integration, we seem to have ironed out most of the issues before >>>> they even showed up. >>>> >>>> At this point, the issues that I truly expect us to see are going to be >>>> capacity related, so if the queues get extra long for a bit, we're >>>> sorry. We're aware that it's a possibility with a change of this >>>> magnitude. We're going to be watching very carefully and doing what we >>>> can to tune things better. >>>> >>>> -Andy- >>>> >>>> On 06/26/2016 09:46 AM, Andrew Grimberg wrote: >>>> > Status update: >>>> > >>>> > Disk management has been completed. Jenkins is online but we're still >>>> > working through the changes that need to happen after Jenkins is >>>> running >>>> > again. >>>> > >>>> > As such Jenkins is going to remain in the non-processing 'going to >>>> shut >>>> > down' mode until we have finished our changes. >>>> > >>>> > Current estimate still puts us at ~12:00 PDT before we reopen Jenkins >>>> > for proper service. >>>> > >>>> > -Andy- >>>> > >>>> > On 06/26/2016 06:04 AM, Andrew Grimberg wrote: >>>> >> Greetings folks, >>>> >> >>>> >> Just an update on the outage. Yes, we're still down, but we're >>>> finally >>>> >> into the home stretch of disk changes before we can restart Jenkins >>>> and >>>> >> then apply the needed job changes related to the Jenkins updates. >>>> >> >>>> >> With the present rate at which the related disk changes are >>>> happening I >>>> >> am presently anticipating that we'll have it back online by 12:00 PDT >>>> >> today (6 hours out). >>>> >> >>>> >> My apologies for the longer outage duration than originally planned! >>>> >> -Andy- >>>> >> >>>> >> On 06/25/2016 04:55 PM, Andrew Grimberg wrote: >>>> >>> Original window is about to close but we still haven't completed the >>>> >>> migration. From the look of things we're about 1 - 2 hours away >>>> from the >>>> >>> finalized disk sync finishing. Giving how long we've been down and >>>> that >>>> >>> we're likely to see similar lengths of time for an attempt at a >>>> later >>>> >>> date, we're just going to go ahead and power through. >>>> >>> >>>> >>> Sorry for the extended outage! >>>> >>> >>>> >>> -Andy- >>>> >>> >>>> >>> On 06/25/2016 07:45 AM, Andrew Grimberg wrote: >>>> >>>> This work will be starting in 15 minutes. >>>> >>>> >>>> >>>> -Andy- >>>> >>>> >>>> >>>> On 06/23/2016 09:31 AM, Andrew Grimberg wrote: >>>> >>>>> What: The Linux Foundation will be performing the final migration >>>> of the >>>> >>>>> OpenDaylight Jenkins releng silo (aka production silo) to from the >>>> >>>>> Rackspace public cloud to private cloud >>>> >>>>> >>>> >>>>> When: Saturday, June 25, 2016 @ 08:00 - 17:00 PDT (15:00 - 00:00 >>>> UTC) >>>> >>>>> >>>> >>>>> Why: This is the final step in our migration from the Rackspace >>>> public >>>> >>>>> cloud to the private cloud. >>>> >>>>> >>>> >>>>> Impact: The production Jenkins system will be offline for the >>>> duration >>>> >>>>> of the migration. The size of the window is needed to account for >>>> final >>>> >>>>> data synchronization of the current jenkins silo to the new new >>>> one >>>> >>>>> which consists of ~2.3TB of data. >>>> >>>>> >>>> >>>>> Additionally, while we're doing the final disk sync we will take >>>> the >>>> >>>>> time to do needed system updates on other components of the CI >>>> >>>>> infrastructure. As such, there will be some rolling outages of >>>> Gerrit, >>>> >>>>> Nexus, Sonar, and CLM. >>>> >>>>> >>>> >>>>> We will be sending out a note to the lists and #opendaylight IRC >>>> channel >>>> >>>>> on Freenode at the beginning and end of the maintenance. >>>> >>>>> >>>> >>>>> -Andy- >>>> >>>>> >>>> >>>> >>>> >>> >>>> >> >>>> > >>>> >>>> -- >>>> Andrew J Grimberg >>>> Systems Administrator >>>> Release Engineering Team Lead >>>> The Linux Foundation >>>> >>>> >>>> _______________________________________________ >>>> release mailing list >>>> [email protected] >>>> https://lists.opendaylight.org/mailman/listinfo/release >>>> >>>> >>> >>> >>> -- >>> Thanks >>> Anil >>> >> >> >> >> -- >> Thanks >> Anil >> >> _______________________________________________ >> release mailing list >> [email protected] >> https://lists.opendaylight.org/mailman/listinfo/release >> >> >
_______________________________________________ Discuss mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/discuss
