Hi everybody, cleanups done, the master is now using ~240G of disk space (24% of the root partition).
Luca On Sun, Dec 19, 2021 at 9:16 AM Luca Toscano <[email protected]> wrote: > > Hi Kengo, > > Thanks for the detailed explanation! I'll try to do some cleanup > today/tomorrow, I am 100% in favor of having more EBS space if needed > (if backed up regularly), more free space to dedicate to other > projects if needed :) > > Luca > > On Sun, Dec 19, 2021 at 1:46 AM Kengo Seki <[email protected]> wrote: > > > > Thank you for cleaning up the Jenkins master! As you said, it often > > runs short of disk capacity. > > Your suggestion (keeping only recent build results of actively > > developing/maintaining branches and discarding others) > > sounds reasonable to me, because all of our release artifacts are on > > ASF's distribution server and S3, > > and we can also rerun the Jenkins job, if needed. > > > > On the other hand, after migrating to the new account, the situation > > will get better. > > We've used m3.xlarge (4vCPUs, 15GiB mem) for the Jenkins master, which > > costs $0.308/h. > > If we replace it with m6a.xlarge (4vCPUs, 16GiB mem) in the new > > account, it costs only $0.1728/h, so we can save $97.344/mo. > > The price of gp3 EBS volume is $0.08/mo for 1GB, so I'm estimating we > > can add 1.2TB extra capacity to the master node. > > > > Kengo Seki <[email protected]> > > > > On Sat, Dec 18, 2021 at 6:45 PM Luca Toscano <[email protected]> wrote: > > > > > > Hi Kengo! > > > > > > Thanks a lot for all the info, I don't have time this weekend too, so > > > I'll let you do the work without messing up the current environment. I > > > am more than happy to help if you need anything next week, feel free > > > to drop me an email in case! > > > > > > One thing that I'd like to sort out with you and others, before > > > proceeding, is the retention of the Jenkins build logs/files/etc.. We > > > have a huge partition on the master instance at the moment, that I > > > believe is/was filled up by old files that we don't really use. For > > > example, can we clean up old build logs after we cut a release? > > > Ideally, in my opinion, if those are not needed afterwards we could: > > > - Keep 1 or 2 recent builds for Trunk (for all the various jobs). The > > > job that builds packages in trunk, for example, uses a ton of GBs for > > > every round of builds (so every week). > > > - Clean up all the rest (even manually, build logs for 1.5.0, 3.0.0, > > > 1.3.0, etc..) > > > > > > I already cleaned up a bit the other week since the master instance's > > > partition was filled up, I'd be happy to finish the work this weekend > > > if you agree :) > > > > > > Lemme know! > > > > > > Thanks, > > > > > > Luca > > > > > > On Fri, Dec 17, 2021 at 2:13 PM Kengo Seki <[email protected]> wrote: > > > > > > > > Hi Luca, thank you for working on them! (and thank you for helping him, > > > > Olaf!) > > > > Let me share our current situation about the CI environment. > > > > > > > > Our CI infrastructure is provided by courtesy of AWS, and for > > > > addressing their security request, > > > > we're going to integrate our environment (mainly EC2 instances and > > > > files on S3) > > > > to another AWS account in this month (I'm sorry for being late to > > > > share this information). > > > > So BIGTOP-3612 is very helpful for us, because we can take over the > > > > contents under /home/jenkins > > > > by sharing the EBS snapshot between the old and new accounts. > > > > > > > > On the other hand, BIGTOP-3611 is not necessarily required, because > > > > new EC2 instances > > > > for Jenkins and workers will be launched in the new account within one > > > > or two weeks. > > > > But if you could upgrade Jenkins, it's still be helpful, because we > > > > can check if all of the Jenkins plugins > > > > we're currently using are compatible with the latest version in advance. > > > > > > > > I'm a bit busy until this weekend, so I'm planning to start the > > > > migration work > > > > (copying files between S3 buckets, launching Jenkins and worker nodes, > > > > etc.) next week. > > > > So, if you are going to work on the issues above in this weekend, > > > > would you share the result after that? > > > > > > > > Kengo Seki <[email protected]> > > > > > > > > On Thu, Dec 16, 2021 at 1:30 AM Olaf Flebbe <[email protected]> wrote: > > > > > > > > > > hi luca > > > > > > > > > > we can do the update together in the next days (evening local time in > > > > > eu). > > > > > > > > > > best > > > > > olaf > > > > > > > > > > > Am 15.12.2021 um 09:37 schrieb Luca Toscano > > > > > > <[email protected]>: > > > > > > > > > > > > Hi everybody, > > > > > > > > > > > > Any feedback? I'd like to upgrade, if everybody agrees, Jenkins > > > > > > during > > > > > > the next days. If anybody can review the procedure and give me a > > > > > > +1/-1 > > > > > > I'd be grateful :) > > > > > > Moreover, if anybody wants to be online with me when I do the > > > > > > upgrade > > > > > > it would be really great, so if anything goes wrong there will be > > > > > > more > > > > > > people watching. The upgrade itself shouldn't last long (10/15 mins > > > > > > if > > > > > > everything goes fine). > > > > > > > > > > > > Thanks in advance! > > > > > > > > > > > > Luca > > > > > > > > > > > >> On Sat, Dec 11, 2021 at 10:01 AM Luca Toscano > > > > > >> <[email protected]> wrote: > > > > > >> > > > > > >> Hi everybody, > > > > > >> > > > > > >> I opened a couple of Jiras for Jenkins: > > > > > >> - https://issues.apache.org/jira/browse/BIGTOP-3611 - Upgrade > > > > > >> Jenkins > > > > > >> to the latest upstream > > > > > >> - https://issues.apache.org/jira/browse/BIGTOP-3612 - Add a backup > > > > > >> for > > > > > >> Jenkins' /home/jenkins dir > > > > > >> > > > > > >> I didn't find anything open for these topics, apologies in advance > > > > > >> in > > > > > >> case there is known work in progress. > > > > > >> > > > > > >> Let me know your thoughts :) > > > > > >> > > > > > >> Luca > > > > >
