Re: Increasing Discard Old Builds strategy in CI
Hello, I found the post-action "Delete Workspace when build is done", I updated https://ci-builds.apache.org/job/TomEE/job/master-pull-request/ to check if this can reduce the job size. If this doesn't work I'll rollback the change and keep you updated. [image: image.png] El mar, 26 abr 2022 a las 13:43, Cesar Hernandez () escribió: > I forgot that /home/jenkins/.m2 is in the Agent and what is actually > analyzed for the test results are the surefire files as Richard mentioned: > > /home/jenkins/jenkins-agent/workspace/Tomee/master-pull-request/ > > > I'm +1 on trying a mvn clean as a post step to see if that reduces the > size of the job result without loosing logs. > > > > > El mar, 26 abr 2022 a las 12:52, Cesar Hernandez () > escribió: > >> In the job, after the test results are executed we can add a post-build >> doing a: >> rm -rf /home/jenkins/.m2/ >> >> >> >> >> El mar, 26 abr 2022 a las 12:51, Zowalla, Richard (< >> richard.zowa...@hs-heilbronn.de>) escribió: >> >>> Afaik Jenkins is parsing the surefire report files. The log files might >>> also be important for the creator of the PR, so we would need to keep >>> them too. >>> >>> Might be worth a try, I guess. >>> >>> Gruß >>> Richard >>> >>> Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins: >>> > > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard < >>> > > richard.zowa...@hs-heilbronn.de> wrote: >>> > > >>> > > However, we need to carefully monitor the disk usage / volume used >>> > > as >>> > > INFRA has a problem if our used storage volume per jobs exceeds a >>> > > certain quota. >>> > > >>> > > We occupied 1,5T last week (cumulated over a few weeks) and they >>> > > were a >>> > > bit mad as the Jenkins CI env went out of storage. >>> > >>> > I was going to jump in on that thread and got side tracked. >>> > >>> > I wonder if we can do a `mvn clean` or something similar as a post >>> > build step. Looks like after a build we have 2.8G used of disk space >>> > and once we clean that goes down to 398M. >>> > >>> > The test case results are the obvious trick. I'm not sure if Jenkins >>> > keeps those somewhere else or if we need them to keep living in the >>> > target directories. If they need to stay in the target directories >>> > we can maybe create a script that deletes everything in target/ but >>> > surefire results. >>> > >>> > If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5 >>> > gigabytes. >>> > >>> > Then we could just keep 30 builds regardless of days and it should be >>> > good. >>> > >>> > Thoughts? >>> > >>> > >>> > -David >>> > >>> >> >> >> -- >> Atentamente: >> César Hernández. >> > > > -- > Atentamente: > César Hernández. > -- Atentamente: César Hernández.
Re: Increasing Discard Old Builds strategy in CI
I forgot that /home/jenkins/.m2 is in the Agent and what is actually analyzed for the test results are the surefire files as Richard mentioned: /home/jenkins/jenkins-agent/workspace/Tomee/master-pull-request/ I'm +1 on trying a mvn clean as a post step to see if that reduces the size of the job result without loosing logs. El mar, 26 abr 2022 a las 12:52, Cesar Hernandez () escribió: > In the job, after the test results are executed we can add a post-build > doing a: > rm -rf /home/jenkins/.m2/ > > > > > El mar, 26 abr 2022 a las 12:51, Zowalla, Richard (< > richard.zowa...@hs-heilbronn.de>) escribió: > >> Afaik Jenkins is parsing the surefire report files. The log files might >> also be important for the creator of the PR, so we would need to keep >> them too. >> >> Might be worth a try, I guess. >> >> Gruß >> Richard >> >> Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins: >> > > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard < >> > > richard.zowa...@hs-heilbronn.de> wrote: >> > > >> > > However, we need to carefully monitor the disk usage / volume used >> > > as >> > > INFRA has a problem if our used storage volume per jobs exceeds a >> > > certain quota. >> > > >> > > We occupied 1,5T last week (cumulated over a few weeks) and they >> > > were a >> > > bit mad as the Jenkins CI env went out of storage. >> > >> > I was going to jump in on that thread and got side tracked. >> > >> > I wonder if we can do a `mvn clean` or something similar as a post >> > build step. Looks like after a build we have 2.8G used of disk space >> > and once we clean that goes down to 398M. >> > >> > The test case results are the obvious trick. I'm not sure if Jenkins >> > keeps those somewhere else or if we need them to keep living in the >> > target directories. If they need to stay in the target directories >> > we can maybe create a script that deletes everything in target/ but >> > surefire results. >> > >> > If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5 >> > gigabytes. >> > >> > Then we could just keep 30 builds regardless of days and it should be >> > good. >> > >> > Thoughts? >> > >> > >> > -David >> > >> > > > -- > Atentamente: > César Hernández. > -- Atentamente: César Hernández.
Re: Increasing Discard Old Builds strategy in CI
In the job, after the test results are executed we can add a post-build doing a: rm -rf /home/jenkins/.m2/ El mar, 26 abr 2022 a las 12:51, Zowalla, Richard (< richard.zowa...@hs-heilbronn.de>) escribió: > Afaik Jenkins is parsing the surefire report files. The log files might > also be important for the creator of the PR, so we would need to keep > them too. > > Might be worth a try, I guess. > > Gruß > Richard > > Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins: > > > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard < > > > richard.zowa...@hs-heilbronn.de> wrote: > > > > > > However, we need to carefully monitor the disk usage / volume used > > > as > > > INFRA has a problem if our used storage volume per jobs exceeds a > > > certain quota. > > > > > > We occupied 1,5T last week (cumulated over a few weeks) and they > > > were a > > > bit mad as the Jenkins CI env went out of storage. > > > > I was going to jump in on that thread and got side tracked. > > > > I wonder if we can do a `mvn clean` or something similar as a post > > build step. Looks like after a build we have 2.8G used of disk space > > and once we clean that goes down to 398M. > > > > The test case results are the obvious trick. I'm not sure if Jenkins > > keeps those somewhere else or if we need them to keep living in the > > target directories. If they need to stay in the target directories > > we can maybe create a script that deletes everything in target/ but > > surefire results. > > > > If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5 > > gigabytes. > > > > Then we could just keep 30 builds regardless of days and it should be > > good. > > > > Thoughts? > > > > > > -David > > > -- Atentamente: César Hernández.
Re: Increasing Discard Old Builds strategy in CI
Afaik Jenkins is parsing the surefire report files. The log files might also be important for the creator of the PR, so we would need to keep them too. Might be worth a try, I guess. Gruß Richard Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins: > > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard < > > richard.zowa...@hs-heilbronn.de> wrote: > > > > However, we need to carefully monitor the disk usage / volume used > > as > > INFRA has a problem if our used storage volume per jobs exceeds a > > certain quota. > > > > We occupied 1,5T last week (cumulated over a few weeks) and they > > were a > > bit mad as the Jenkins CI env went out of storage. > > I was going to jump in on that thread and got side tracked. > > I wonder if we can do a `mvn clean` or something similar as a post > build step. Looks like after a build we have 2.8G used of disk space > and once we clean that goes down to 398M. > > The test case results are the obvious trick. I'm not sure if Jenkins > keeps those somewhere else or if we need them to keep living in the > target directories. If they need to stay in the target directories > we can maybe create a script that deletes everything in target/ but > surefire results. > > If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5 > gigabytes. > > Then we could just keep 30 builds regardless of days and it should be > good. > > Thoughts? > > > -David > smime.p7s Description: S/MIME cryptographic signature
Re: Increasing Discard Old Builds strategy in CI
> On Apr 26, 2022, at 10:47 AM, Zowalla, Richard > wrote: > > However, we need to carefully monitor the disk usage / volume used as > INFRA has a problem if our used storage volume per jobs exceeds a > certain quota. > > We occupied 1,5T last week (cumulated over a few weeks) and they were a > bit mad as the Jenkins CI env went out of storage. I was going to jump in on that thread and got side tracked. I wonder if we can do a `mvn clean` or something similar as a post build step. Looks like after a build we have 2.8G used of disk space and once we clean that goes down to 398M. The test case results are the obvious trick. I'm not sure if Jenkins keeps those somewhere else or if we need them to keep living in the target directories. If they need to stay in the target directories we can maybe create a script that deletes everything in target/ but surefire results. If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5 gigabytes. Then we could just keep 30 builds regardless of days and it should be good. Thoughts? -David smime.p7s Description: S/MIME cryptographic signature
Re: Increasing Discard Old Builds strategy in CI
Thank you for the context Richard, My proposal is for master PR's only since I think this is where most of the activity is going on at the moment. We occupied 1,5T last week In that case, I think a more conservative proposal would be: Days to keep builds: 8 Max # of builds to keep: 15 El mar, 26 abr 2022 a las 11:47, Zowalla, Richard (< richard.zowa...@hs-heilbronn.de>) escribió: > Hi Cesar, > > I have no problem with increasing the values for the discard strategy. > It makes sense to keep it a bit longer for PRs. > > However, we need to carefully monitor the disk usage / volume used as > INFRA has a problem if our used storage volume per jobs exceeds a > certain quota. > > We occupied 1,5T last week (cumulated over a few weeks) and they were a > bit mad as the Jenkins CI env went out of storage. > > Gruß > Richard > > > Am Dienstag, dem 26.04.2022 um 11:39 -0600 schrieb Cesar Hernandez: > > Hello, > > > > Our current Discard old builds Strategy in CI for PR's for the master > > branch is [1]: > > Days to keep builds: 2 > > Max # of builds to keep: 3 > > > > Is there any objection if I update this to at least: > > Days to keep builds: 15 > > Max # of builds to keep: 30 > > > > Currently, if one opens a PR on a Friday and tries to catch up on > > mid-Monday next week, the PR results are gone. > > The above proposal will also give us a bit more room for more context > > on > > how PR's and past builds have evolved, especially when trying to > > chase and > > resolve random test failures. > > > > > > [1] > > https://ci-builds.apache.org/job/Tomee/job/master-pull-request > -- Atentamente: César Hernández.
Re: Increasing Discard Old Builds strategy in CI
Hi Cesar, I have no problem with increasing the values for the discard strategy. It makes sense to keep it a bit longer for PRs. However, we need to carefully monitor the disk usage / volume used as INFRA has a problem if our used storage volume per jobs exceeds a certain quota. We occupied 1,5T last week (cumulated over a few weeks) and they were a bit mad as the Jenkins CI env went out of storage. Gruß Richard Am Dienstag, dem 26.04.2022 um 11:39 -0600 schrieb Cesar Hernandez: > Hello, > > Our current Discard old builds Strategy in CI for PR's for the master > branch is [1]: > Days to keep builds: 2 > Max # of builds to keep: 3 > > Is there any objection if I update this to at least: > Days to keep builds: 15 > Max # of builds to keep: 30 > > Currently, if one opens a PR on a Friday and tries to catch up on > mid-Monday next week, the PR results are gone. > The above proposal will also give us a bit more room for more context > on > how PR's and past builds have evolved, especially when trying to > chase and > resolve random test failures. > > > [1] > https://ci-builds.apache.org/job/Tomee/job/master-pull-request smime.p7s Description: S/MIME cryptographic signature
Increasing Discard Old Builds strategy in CI
Hello, Our current Discard old builds Strategy in CI for PR's for the master branch is [1]: Days to keep builds: 2 Max # of builds to keep: 3 Is there any objection if I update this to at least: Days to keep builds: 15 Max # of builds to keep: 30 Currently, if one opens a PR on a Friday and tries to catch up on mid-Monday next week, the PR results are gone. The above proposal will also give us a bit more room for more context on how PR's and past builds have evolved, especially when trying to chase and resolve random test failures. [1] https://ci-builds.apache.org/job/Tomee/job/master-pull-request -- Atentamente: César Hernández.