Discard old builds is very likely the problem. If a tag build is removed from the history by "Discard old builds" then future searches for the that tag (or the SHA-1 of the commit associated with the tag) will fail and it will build again.
On Mon, Jan 6, 2020 at 5:18 PM Eric Pierson <[email protected]> wrote: > Thank you for your response, Mark! > > I have not tried to run any scripts to prune history as I see described in > JENKINS-19022. There are a lot of comments in this one and I am still > re-reading them to be sure I am not missing something else relevant folks > discovered along the way. > > I do have "Discard old builds" enabled with Strategy "Log Rotation" and > "Max # of builds to keep" = 10. Forgive me for omitting the fact that I > use Discard old builds initially. > > Reading more about "Discard old builds": > > - Build count: discard the *oldest build* when a certain number of > builds already exist > - Note that Jenkins does not discard items immediately when this > configuration is updated, or as soon as any of the configured values are > exceeded; *these rules are evaluated each time a build of this project > completes.* > > I am now focusing on the "Discard old builds" settings and build.xml > content, and what may be happening to the build history when a build > completes. I have one job that has only run a total of 7 times that had > the issue on builds #3 and #4, and the example I initially provided has > only run a total of 12 times and we see the issue on multiple builds there, > too. So while there are indeed many tags in the repo, the build history is > not "large" yet. > > I have another Jenkins environment I used for a while where I did not have > this issue. It had the same settings (including "Discard old builds"), but > when I was running jobs through this environment on a regular basis, > Jenkins and plugins would have been at older versions. > > To hone in on git plugin versions just in the change they are relevant, I > compared my build.xml: > > > New Environment > <hudson.plugins.git.util.BuildData plugin="[email protected]"> > <marked plugin="[email protected]"> > <revision plugin="[email protected]"> > > Old Environment > <hudson.plugins.git.util.BuildData plugin="[email protected]"> > <marked plugin="[email protected]"> > <revision plugin="[email protected]"> > > > --------------- > > Given this additional information, please let me know if you have any new > thoughts or if you feel I should file a bug report (for perhaps the > logRotate or BuildDiscarder area?) as I continue to research and test. > > Thank you again for your help and everything you do for the Jenkins > community! > > -Eric > > > > On Fri, Jan 3, 2020 at 8:38 PM Mark Waite <[email protected]> > wrote: > >> >> >> On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson <[email protected]> wrote: >> >>> I am troubleshooting a scenario where jobs with a wildcard tag refspec >>> (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and >>> rebuild a successfully built tag referencing the same commit. >>> >>> When a rerun occurs, prior job runs for the tag are no longer present in >>> the Git Build History for the job. >>> >>> >> That sounds to me as though you are relying on the JENKINS-19022 memory >> bloat bug to retain history of more jobs in the BuildData than are actually >> in the build history. >> >> If someone executed one of the scripts that is mentioned in JENKINS-19022 >> to clean the excess BuildData from the history, that seems like it might >> cause the SHA-1's referenced by some of the tags to be removed from the >> history. >> >> If build history were being removed based on a specific number of builds, >> then it also could be that the history entry which included that tag was >> removed from the list, and thus was no longer visible. >> >> Mark Waite >> >> >>> I am seeking help to further investigate and trace polling activity to >>> try to determine what changes are being detected that trigger the job to >>> run again, or learn if i should submit this as a bug report for assistance >>> instead. >>> >>> Thanks, everyone! >>> >>> >>> --------------- >>> Existing Bugs Found: >>> >>> This was the most relative bug I could find, yet my situation is >>> different. >>> >>> >>> https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true >>> --------------- >>> >>> >>> *My Environment:* >>> >>> Jenkins 2.176.3 on traditional VMs (not containers) >>> GitHub Plugin 1.29.5 >>> Git Plugin 4.0.0 >>> >>> >>> *Steps to reproduce:* >>> >>> 1. Add a webhook to a GitHub repo which sends the following events to >>> Jenkins: Pushes, Branch or tag creation, Releases. >>> >>> 2. Create two “Pipeline” jobs in Jenkins, both referencing the same >>> GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. >>> >>> *NOTE: JJB is used to create the jobs (jobs are not copied).* >>> >>> >>> >>> Settings for job 1 (master branch job): >>> Do not allow concurrent builds >>> GitHub hook trigger for GITScm polling >>> Pipeline script from SCM: Git >>> Repo ID: origin >>> Refspec: +refs/heads/*:refs/remotes/origin/* >>> Branch Specifier: refs/heads/master >>> Additional Behaviors: Wipe out repository & force clone >>> Lightweight checkout: false >>> >>> >>> >>> Settings for job 2 (tag job): >>> Do not allow concurrent builds >>> GitHub hook trigger for GITScm polling >>> Pipeline script from SCM: Git >>> Repo ID: origin >>> Refspec: +refs/tags/*:refs/remotes/origin/tags/* >>> Branch Specifier: */tags/* >>> Additional Behaviors: Wipe out repository & force clone >>> Lightweight checkout: false >>> >>> >>> 3. When commits are merged to the master branch of the repo, the >>> webhook fires and Job 1 executes. >>> >>> 4. When new releases (tags) are published, the webhook fires and Job 2 >>> executes for the most recent (new) tag. >>> >>> (Things have been running smoothly for me with this type of setup for a >>> couple years.) >>> >>> 5. Occasionally there is a situation with jobs configured like job 2, >>> specifically: >>> >>> - New commits are pushed to master which triggers the webhook to >>> Jenkins. >>> - Job 1 polls and detects the new commits to master and runs. >>> - Job 2 polls and detects changes even though no tags have been >>> created and previously built tags still reference the same commit. >>> - Job 2 then runs and rebuilds the last tag it had already built >>> successfully. >>> - The polling log shows last built revision is the same commit and >>> tag, no diff is performed, but changes are found. >>> - The changes page for the job shows no changes. >>> >>> >>> Observations when this occurs: >>> >>> - Let's say the tag has been built 4 times. We will have a polling >>> log and change logs for the first job run that initially built the tag, >>> then jobs 2 and 3 will have no polling log on the filesystem but will >>> have >>> an empty changelog0.xml, then the last build of the tag has a polling log >>> and change log files. >>> - When a tag is being re-built, when you look at Git Build Data for >>> the job rebuilding that tag, all prior job runs that built that tag >>> successfully are missing from the history. >>> - The problem does not always happen in my environment. It can >>> happen for a repo one day then not happen the next. Perhaps I am simply >>> overlooking some activity in the repo that results in polling detecting >>> changes with my wildcard tag specifier. >>> - Workspaces appear to be intact as required for the git plugin to >>> perform a wildcard tag poll (post-clone). There are no indications of >>> workspaces needing to be created as job runs start. >>> - The same build node (same workspace) can be used for all job runs >>> and the issue can still occur. >>> >>> >>> >>> *Example polling log and Git Build Data for a re-run:* >>> >>> Started on Dec 30, 2019 7:54:51 AM >>> Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on >>> Mon Dec 30 07:54:51 EST 2019 >>> Using strategy: Default >>> [poll] Last Built Revision: Revision >>> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >>> refs/tags/v2.3.0) >>> using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913 >>> > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10 >>> Fetching changes from the remote Git repositories >>> > /data/git-client/bin/git config remote.origin.url >>> https://github.com/repo.git # timeout=10 >>> Fetching upstream changes from https://github.com/repo.git >>> > /data/git-client/bin/git --version # timeout=10 >>> using GIT_ASKPASS to set credentials xxxx Service Account >>> > /data/git-client/bin/git fetch --tags --force --progress -- >>> https://github.com/repo.git +refs/tags/*:refs/remotes/origin/tags/* # >>> timeout=10 >>> Polling for changes in >>> Seen branch in repository origin/2.2.0 >>> Seen branch in repository origin/2.3.0 >>> Seen branch in repository origin/2.4.0 >>> Seen branch in repository origin/gh-pages >>> Seen branch in repository origin/master >>> Seen branch in repository origin/openshift-migration >>> Seen branch in repository origin/rhel-build >>> Seen branch in repository origin/tags/v0.10.0 >>> Seen branch in repository origin/tags/v0.3 >>> Seen branch in repository origin/tags/v0.4 >>> Seen branch in repository origin/tags/v0.7.0 >>> Seen branch in repository origin/tags/v0.7.1 >>> Seen branch in repository origin/tags/v0.8.0 >>> Seen branch in repository origin/tags/v0.9.0 >>> Seen branch in repository origin/tags/v0.9.1 >>> Seen branch in repository origin/tags/v1.0.0 >>> Seen branch in repository origin/tags/v1.0.0-secure >>> Seen branch in repository origin/tags/v1.1.0 >>> Seen branch in repository origin/tags/v1.2.0 >>> Seen branch in repository origin/tags/v1.3.0 >>> Seen branch in repository origin/tags/v1.3.1 >>> Seen branch in repository origin/tags/v1.4.0 >>> Seen branch in repository origin/tags/v1.5.0 >>> Seen branch in repository origin/tags/v1.6.0 >>> Seen branch in repository origin/tags/v1.6.1 >>> Seen branch in repository origin/tags/v1.7.0 >>> Seen branch in repository origin/tags/v1.7.1 >>> Seen branch in repository origin/tags/v1.7.2 >>> Seen branch in repository origin/tags/v1.8.0 >>> Seen branch in repository origin/tags/v1.9.0 >>> Seen branch in repository origin/tags/v2.0.0 >>> Seen branch in repository origin/tags/v2.1.0 >>> Seen branch in repository origin/tags/v2.2.0 >>> Seen branch in repository origin/tags/v2.3.0 >>> Seen 34 remote branches >>> > /data/git-client/bin/git show-ref --tags -d # timeout=10 >>> Using strategy: Default >>> [poll] Last Built Revision: Revision >>> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >>> refs/tags/v2.3.0) >>> Done. Took 0.53 sec >>> Changes found >>> >>> --------------------------------- >>> >>> Git Build Data (for Build #9) >>> Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc >>> origin/tags/v2.4.0 >>> refs/tags/v2.4.0 >>> Built Branches >>> refs/tags/v2.1.0: Build #4 of Revision >>> 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, >>> refs/tags/v2.1.0) >>> origin/tags/v2.2.0: Build #5 of Revision >>> 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, >>> refs/tags/v2.2.0) >>> refs/tags/v2.2.0: Build #5 of Revision >>> 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, >>> refs/tags/v2.2.0) >>> origin/tags/v2.1.0: Build #4 of Revision >>> 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, >>> refs/tags/v2.1.0) >>> origin/tags/v2.0.0: Build #2 of Revision >>> 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, >>> refs/tags/v2.0.0) >>> refs/tags/v2.4.0: Build #9 of Revision >>> 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, >>> refs/tags/v2.4.0) >>> refs/tags/v2.3.0: Build #8 of Revision >>> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >>> refs/tags/v2.3.0) >>> origin/tags/v1.0.0-secure: Build #1 of Revision >>> 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, >>> refs/tags/v1.0.0-secure) >>> origin/tags/v2.4.0: Build #9 of Revision >>> 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, >>> refs/tags/v2.4.0) >>> refs/tags/v1.0.0-secure: Build #1 of Revision >>> 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, >>> refs/tags/v1.0.0-secure) >>> refs/tags/v2.0.0: Build #2 of Revision >>> 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, >>> refs/tags/v2.0.0) >>> origin/tags/v2.3.0: Build #8 of Revision >>> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >>> refs/tags/v2.3.0) >>> >>> NOTE: >>> Build #6 was the initial build of refs/tags/v2.3.0 >>> Build #7 was the second rerun of refs/tags/v2.3.0 >>> Build #8 was the last rerun of refs/tags/v2.3.0 >>> Build #9 was for refs/tags/v2.4.0 >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com >>> <https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Thanks! >> Mark Waite >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Jenkins Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jenkinsci-users/_JJM5DZoPvA/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Thanks! Mark Waite -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGNKPNDDfBNKwbN22%2BVK18VFSmT6%2BWYD1syv95Ucs98vg%40mail.gmail.com.
