We recently developed an internal script to clean and report (html email) any pipeline instances that were cleaned, and how much space was recovered. The gocd-janitor utility was a nice reference for what we wanted. We ended up creating a python script that only removes "failed" builds, and keeps the console.log files around. For us, the logs are still valuable to look at and do not take up a considerable amount of space. We also had a more specific definition of what a "failed" build was. There was also one very obvious space saver that we noticed. Removing "orphaned" pipeline artifact folders. This might not occur very frequently, but if you do remove a pipeline config at any point, you will also want to remove the pipeline folder on disk, otherwise it just sits around with not references.
On Tuesday, November 22, 2016 at 10:20:56 AM UTC-8, Ashwanth Kumar wrote: > > Leo, I took a quick glance at the repo, it's simple and seems to do it's > job. Thanks for the contribution. > > One feedback though, the approach of deleting anything older than 180 / > 100 days would work fine if you're doing just CI builds. Cases when you're > taking artifacts to various pipelines using Go's Artifact dependency - I > would recommend to give gocd-janitor > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fashwanthkumar%2Fgocd-janitor&sa=D&sntz=1&usg=AFQjCNF-RVxIp5H1ZwNRfyQAOU6twPyGxA> > > a shot. It looks at all the dependences of each pipeline version and > maintains the respective versions accordingly. > > I've personally been part of various solutions to this problem over the > past 4+ years and gocd-janitor seems to be the best that works out for us > so far. > > > On Tue, Nov 22, 2016 at 10:38 PM, Leo Keuken <[email protected] > <javascript:>> wrote: > >> We were facing pretty much facing the same issue everyone has come across >> by now I think: constant alerting on full disk on the vm where the >> go-server lives. Though there is a lot to say for project management of >> stuff that is being run inside the pipelines and what it saves onto the >> server in a typical build (e.g. massive test reports from certain suites), >> all in all it will fill up slowly as no build is ever discarded. With many >> projects a real headache! >> >> Before looking on groups or the www extensively I just went to work and >> re-invent the wheel (of course) and I wrote a small shell script to get rid >> of the old builds that usually are obsolete in a fast progressing >> environment. I've put it on github: >> https://github.com/LeoK80/go-server-cleanup . Feel free to fork it and >> make fit for your own purposes or just use it out-of-the-box. Readme will >> tell you all you need to know about its use. >> >> Defaults: >> - go-server artifacts directory '/var/lib/go-server/artifacts/pipelines' >> - keep anything younger than 180 days >> - never delete from a pipeline when there is 15 or less builds present >> There is parameter input provided for retention & minimum of builds to >> keep >> >> >> On Wednesday, April 29, 2015 at 10:20:20 PM UTC+1, Jason D wrote: >>> >>> Due to problems the default artifact cleanup algorithm of GO can >>> potentially cause, we have decided to turn it off and write our own. >>> >>> Our simple script currently just cycles through the artifacts and >>> deletes all but the latest xx (configurable) for every pipeline. Very >>> simple place to start and it works well. >>> >>> We'd now like to add a bit more sophistication which brings me to the >>> questions I have for you all. >>> >>> Is there a programatic way (via the GO api, I assume) to tell if a >>> particular artifact set has been successful or not, given only the info >>> available from the file system for artifacts? In the next iteration we'd >>> like to tighten it up a bit and save, for instance, the last xx artifact >>> sets that were the result of a pipeline that completed successfully, or >>> similar. >>> >>> If others have ideas on how you have managed this issue, I'd be grateful >>> for your feedback. >>> >>> Thanks. >>> jason >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Ashwanth Kumar / ashwanthkumar.in > > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
