Ok, that job doesn't/wouldn't work, since 'sudo' doesn't work from a job. It asks for a password and fails.
I tried modifying that job to do other cleaning, but I've not succeeded with any variation like su jenkins -c apt-get ... With the default permissions nothing sensible can be done. Zoltan was blocked on his SDK building on this, so I went to #webops and luckily found pjdc active as there is no vanguard in my morning. He "twiddled the reserved block counts to match policy", and additionally ran both the commands I suggested and pbuilder --clean (since nothing was building). 5.2G free on / -> we should live another day (or week). All seems good currently. -Timo On Tue, Apr 29, 2014 at 7:04 AM, Timo Jyrinki <[email protected]> wrote: > pbuilder --clean should work but it's indeed the biggest weapon > available. I'd rather go for removing the older directories based on > timestamps. > > Another place that will eat all the disk space eventually is > pbuilder's apt-cache, and I used the following to clean that the last > time: > mkdir /var/cache/pbuilder/trusty-amd64/aptcache/partial > apt-get -o Dir::Cache::archives=/var/cache/pbuilder/trusty-amd64/aptcache > autoclean > > -Timo > > > On Tue, Apr 29, 2014 at 12:44 AM, Francis Ginther > <[email protected]> wrote: >> Mathieu suggested using 'pbuilder --clean' [1], which in my local >> testing appears to do the trick (and shouldn't require IS intervention >> if pbuilder is on the sudo list). The big thing to note is that you >> don't want to run this while a build is in progress (I don't think >> pbuilder checks for builds in progress). >> >> I've created a 'pbuilder-clean' job to perform this, but have set it >> to disabled and have not executed it. If you decide this is the right >> action, all you need to do is enable the job and run it. >> >> [1] - pbuilder man page: >> --clean >> Cleans up the directory specified by the configuration >> BUILD‐ >> PLACE and APTCACHE specified in /etc/pbuilderrc >> >> Francis >> >> On Mon, Apr 28, 2014 at 4:23 PM, Łukasz 'sil2100' Zemczak >> <[email protected]> wrote: >>> Hi Francis >>> >>> On 28.04.2014 22:19, Francis Ginther wrote: >>>> Builds are now failing due to insufficient disk space [1]. I've >>>> determined that 60% of / is consumed by /var/cache/pbuilder/build [2] >>>> and most of those directories have older timestamps. My theory is that >>>> these were created by failed or aborted builds and were not cleaned >>>> up. >>> >>> Argh, that again... Yeah, we had something similar on the previous >>> jenkins on canonistack. We had to clean up the pbuilder cache then as >>> well then through ssh, I can't remember if we thought up some long-term >>> remedy back then. >>> >>>> I would like to request that any directory with a timestamp > 1 day be >>>> purged by IS. However, I don't want to request deletion from IS >>>> without someone to confirm this proposal. If someone with a little >>>> more ci-train knowledge can confirm, I'll send the rt request. >>>> >>>> Long term I recommend a cleanup job to run at least daily to purge the >>>> old build dirs. It will need sudo permissions as these directories are >>>> owned by root. >>>> >>>> For the short term, I recommend only executing one build at a time (at >>>> least through the cow build portion of the build). >>>> >>>> [1] - https://ci-train.ubuntu.com/job/landing-002-1-build/41/console >>>> - mktemp: failed to create directory via template >>>> '/tmp/debsign.XXXXXXXX': No space left on device >>>> [2] - https://ci-train.ubuntu.com/job/fginther-test/9/console >>>> >>>> Francis >>>> >>> >>> I wouldn't try to make any changes without Didier around - he should be >>> back on Wednesday. Especially that we don't have ssh access to the >>> machine anymore so we can't really see how the directory structure looks >>> like there right now. >>> >>> But let's talk tomorrow some more. >>> >>> Best regards, >>> >>> -- >>> Łukasz 'sil2100' Zemczak >>> [email protected] >>> www.canonical.com >> >> >> >> -- >> Francis Ginther >> Canonical - Ubuntu Engineering - Continuous Integration Team -- Mailing list: https://launchpad.net/~canonical-ci-engineering Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ci-engineering More help : https://help.launchpad.net/ListHelp
