Hi Kengo, Thanks for the detailed explanation! I'll try to do some cleanup today/tomorrow, I am 100% in favor of having more EBS space if needed (if backed up regularly), more free space to dedicate to other projects if needed :)
Luca On Sun, Dec 19, 2021 at 1:46 AM Kengo Seki <[email protected]> wrote: > > Thank you for cleaning up the Jenkins master! As you said, it often > runs short of disk capacity. > Your suggestion (keeping only recent build results of actively > developing/maintaining branches and discarding others) > sounds reasonable to me, because all of our release artifacts are on > ASF's distribution server and S3, > and we can also rerun the Jenkins job, if needed. > > On the other hand, after migrating to the new account, the situation > will get better. > We've used m3.xlarge (4vCPUs, 15GiB mem) for the Jenkins master, which > costs $0.308/h. > If we replace it with m6a.xlarge (4vCPUs, 16GiB mem) in the new > account, it costs only $0.1728/h, so we can save $97.344/mo. > The price of gp3 EBS volume is $0.08/mo for 1GB, so I'm estimating we > can add 1.2TB extra capacity to the master node. > > Kengo Seki <[email protected]> > > On Sat, Dec 18, 2021 at 6:45 PM Luca Toscano <[email protected]> wrote: > > > > Hi Kengo! > > > > Thanks a lot for all the info, I don't have time this weekend too, so > > I'll let you do the work without messing up the current environment. I > > am more than happy to help if you need anything next week, feel free > > to drop me an email in case! > > > > One thing that I'd like to sort out with you and others, before > > proceeding, is the retention of the Jenkins build logs/files/etc.. We > > have a huge partition on the master instance at the moment, that I > > believe is/was filled up by old files that we don't really use. For > > example, can we clean up old build logs after we cut a release? > > Ideally, in my opinion, if those are not needed afterwards we could: > > - Keep 1 or 2 recent builds for Trunk (for all the various jobs). The > > job that builds packages in trunk, for example, uses a ton of GBs for > > every round of builds (so every week). > > - Clean up all the rest (even manually, build logs for 1.5.0, 3.0.0, > > 1.3.0, etc..) > > > > I already cleaned up a bit the other week since the master instance's > > partition was filled up, I'd be happy to finish the work this weekend > > if you agree :) > > > > Lemme know! > > > > Thanks, > > > > Luca > > > > On Fri, Dec 17, 2021 at 2:13 PM Kengo Seki <[email protected]> wrote: > > > > > > Hi Luca, thank you for working on them! (and thank you for helping him, > > > Olaf!) > > > Let me share our current situation about the CI environment. > > > > > > Our CI infrastructure is provided by courtesy of AWS, and for > > > addressing their security request, > > > we're going to integrate our environment (mainly EC2 instances and files > > > on S3) > > > to another AWS account in this month (I'm sorry for being late to > > > share this information). > > > So BIGTOP-3612 is very helpful for us, because we can take over the > > > contents under /home/jenkins > > > by sharing the EBS snapshot between the old and new accounts. > > > > > > On the other hand, BIGTOP-3611 is not necessarily required, because > > > new EC2 instances > > > for Jenkins and workers will be launched in the new account within one > > > or two weeks. > > > But if you could upgrade Jenkins, it's still be helpful, because we > > > can check if all of the Jenkins plugins > > > we're currently using are compatible with the latest version in advance. > > > > > > I'm a bit busy until this weekend, so I'm planning to start the migration > > > work > > > (copying files between S3 buckets, launching Jenkins and worker nodes, > > > etc.) next week. > > > So, if you are going to work on the issues above in this weekend, > > > would you share the result after that? > > > > > > Kengo Seki <[email protected]> > > > > > > On Thu, Dec 16, 2021 at 1:30 AM Olaf Flebbe <[email protected]> wrote: > > > > > > > > hi luca > > > > > > > > we can do the update together in the next days (evening local time in > > > > eu). > > > > > > > > best > > > > olaf > > > > > > > > > Am 15.12.2021 um 09:37 schrieb Luca Toscano <[email protected]>: > > > > > > > > > > Hi everybody, > > > > > > > > > > Any feedback? I'd like to upgrade, if everybody agrees, Jenkins during > > > > > the next days. If anybody can review the procedure and give me a +1/-1 > > > > > I'd be grateful :) > > > > > Moreover, if anybody wants to be online with me when I do the upgrade > > > > > it would be really great, so if anything goes wrong there will be more > > > > > people watching. The upgrade itself shouldn't last long (10/15 mins if > > > > > everything goes fine). > > > > > > > > > > Thanks in advance! > > > > > > > > > > Luca > > > > > > > > > >> On Sat, Dec 11, 2021 at 10:01 AM Luca Toscano > > > > >> <[email protected]> wrote: > > > > >> > > > > >> Hi everybody, > > > > >> > > > > >> I opened a couple of Jiras for Jenkins: > > > > >> - https://issues.apache.org/jira/browse/BIGTOP-3611 - Upgrade Jenkins > > > > >> to the latest upstream > > > > >> - https://issues.apache.org/jira/browse/BIGTOP-3612 - Add a backup > > > > >> for > > > > >> Jenkins' /home/jenkins dir > > > > >> > > > > >> I didn't find anything open for these topics, apologies in advance in > > > > >> case there is known work in progress. > > > > >> > > > > >> Let me know your thoughts :) > > > > >> > > > > >> Luca > > > >
