Thank you for cleaning up the Jenkins master! As you said, it often runs short of disk capacity. Your suggestion (keeping only recent build results of actively developing/maintaining branches and discarding others) sounds reasonable to me, because all of our release artifacts are on ASF's distribution server and S3, and we can also rerun the Jenkins job, if needed.
On the other hand, after migrating to the new account, the situation will get better. We've used m3.xlarge (4vCPUs, 15GiB mem) for the Jenkins master, which costs $0.308/h. If we replace it with m6a.xlarge (4vCPUs, 16GiB mem) in the new account, it costs only $0.1728/h, so we can save $97.344/mo. The price of gp3 EBS volume is $0.08/mo for 1GB, so I'm estimating we can add 1.2TB extra capacity to the master node. Kengo Seki <[email protected]> On Sat, Dec 18, 2021 at 6:45 PM Luca Toscano <[email protected]> wrote: > > Hi Kengo! > > Thanks a lot for all the info, I don't have time this weekend too, so > I'll let you do the work without messing up the current environment. I > am more than happy to help if you need anything next week, feel free > to drop me an email in case! > > One thing that I'd like to sort out with you and others, before > proceeding, is the retention of the Jenkins build logs/files/etc.. We > have a huge partition on the master instance at the moment, that I > believe is/was filled up by old files that we don't really use. For > example, can we clean up old build logs after we cut a release? > Ideally, in my opinion, if those are not needed afterwards we could: > - Keep 1 or 2 recent builds for Trunk (for all the various jobs). The > job that builds packages in trunk, for example, uses a ton of GBs for > every round of builds (so every week). > - Clean up all the rest (even manually, build logs for 1.5.0, 3.0.0, > 1.3.0, etc..) > > I already cleaned up a bit the other week since the master instance's > partition was filled up, I'd be happy to finish the work this weekend > if you agree :) > > Lemme know! > > Thanks, > > Luca > > On Fri, Dec 17, 2021 at 2:13 PM Kengo Seki <[email protected]> wrote: > > > > Hi Luca, thank you for working on them! (and thank you for helping him, > > Olaf!) > > Let me share our current situation about the CI environment. > > > > Our CI infrastructure is provided by courtesy of AWS, and for > > addressing their security request, > > we're going to integrate our environment (mainly EC2 instances and files on > > S3) > > to another AWS account in this month (I'm sorry for being late to > > share this information). > > So BIGTOP-3612 is very helpful for us, because we can take over the > > contents under /home/jenkins > > by sharing the EBS snapshot between the old and new accounts. > > > > On the other hand, BIGTOP-3611 is not necessarily required, because > > new EC2 instances > > for Jenkins and workers will be launched in the new account within one > > or two weeks. > > But if you could upgrade Jenkins, it's still be helpful, because we > > can check if all of the Jenkins plugins > > we're currently using are compatible with the latest version in advance. > > > > I'm a bit busy until this weekend, so I'm planning to start the migration > > work > > (copying files between S3 buckets, launching Jenkins and worker nodes, > > etc.) next week. > > So, if you are going to work on the issues above in this weekend, > > would you share the result after that? > > > > Kengo Seki <[email protected]> > > > > On Thu, Dec 16, 2021 at 1:30 AM Olaf Flebbe <[email protected]> wrote: > > > > > > hi luca > > > > > > we can do the update together in the next days (evening local time in eu). > > > > > > best > > > olaf > > > > > > > Am 15.12.2021 um 09:37 schrieb Luca Toscano <[email protected]>: > > > > > > > > Hi everybody, > > > > > > > > Any feedback? I'd like to upgrade, if everybody agrees, Jenkins during > > > > the next days. If anybody can review the procedure and give me a +1/-1 > > > > I'd be grateful :) > > > > Moreover, if anybody wants to be online with me when I do the upgrade > > > > it would be really great, so if anything goes wrong there will be more > > > > people watching. The upgrade itself shouldn't last long (10/15 mins if > > > > everything goes fine). > > > > > > > > Thanks in advance! > > > > > > > > Luca > > > > > > > >> On Sat, Dec 11, 2021 at 10:01 AM Luca Toscano <[email protected]> > > > >> wrote: > > > >> > > > >> Hi everybody, > > > >> > > > >> I opened a couple of Jiras for Jenkins: > > > >> - https://issues.apache.org/jira/browse/BIGTOP-3611 - Upgrade Jenkins > > > >> to the latest upstream > > > >> - https://issues.apache.org/jira/browse/BIGTOP-3612 - Add a backup for > > > >> Jenkins' /home/jenkins dir > > > >> > > > >> I didn't find anything open for these topics, apologies in advance in > > > >> case there is known work in progress. > > > >> > > > >> Let me know your thoughts :) > > > >> > > > >> Luca > > >
