Issue entered and will update this thread as things develop. https://issues.jenkins-ci.org/browse/JENKINS-60717
Thanks, Eric On Thu, Jan 9, 2020 at 7:53 AM Eric Pierson <[email protected]> wrote: > Yes, I intend to write up the issue today for tracking. > > I have been troubleshooting more but unfortunately don't have anything > valuable to add just yet. I will reply to this thread once the issue has > been opened with more details. > > Eric > > On Thu, Jan 9, 2020, 05:49 Beushausen, Christian < > [email protected]> wrote: > >> Hi Eric, Mark, >> >> >> >> Could you keep the list updated? Or have you created a new issue on Jira >> for this? >> We do see this (or a very similar) issue also for at least one job in our >> environment >> >> >> >> Thank you and cheers. >> >> >> >> Mit freundlichen Gruessen/Best regards, >> >> *Christian Beushausen* >> I S&T PD SW SWF >> Interior Systems & Technology >> >> *E-Mail:* [email protected] >> >> >> *From:* [email protected] < >> [email protected]> *On Behalf Of *Mark Waite >> *Sent:* Dienstag, 7. Januar 2020 01:21 >> *To:* Jenkins Users <[email protected]> >> *Subject:* Re: Jobs with a wildcard tag refspec sometimes rebuild tags >> at same commit >> >> >> >> Discard old builds is very likely the problem. If a tag build is removed >> from the history by "Discard old builds" then future searches for the that >> tag (or the SHA-1 of the commit associated with the tag) will fail and it >> will build again. >> >> >> >> On Mon, Jan 6, 2020 at 5:18 PM Eric Pierson <[email protected]> wrote: >> >> Thank you for your response, Mark! >> >> >> >> I have not tried to run any scripts to prune history as I see described >> in JENKINS-19022. There are a lot of comments in this one and I am still >> re-reading them to be sure I am not missing something else relevant folks >> discovered along the way. >> >> >> >> I do have "Discard old builds" enabled with Strategy "Log Rotation" and >> "Max # of builds to keep" = 10. Forgive me for omitting the fact that I >> use Discard old builds initially. >> >> >> >> Reading more about "Discard old builds": >> >> - Build count: discard the *oldest build* when a certain number of >> builds already exist >> - Note that Jenkins does not discard items immediately when this >> configuration is updated, or as soon as any of the configured values are >> exceeded; *these rules are evaluated each time a build of this >> project completes.* >> >> I am now focusing on the "Discard old builds" settings and build.xml >> content, and what may be happening to the build history when a build >> completes. I have one job that has only run a total of 7 times that had >> the issue on builds #3 and #4, and the example I initially provided has >> only run a total of 12 times and we see the issue on multiple builds there, >> too. So while there are indeed many tags in the repo, the build history is >> not "large" yet. >> >> >> >> I have another Jenkins environment I used for a while where I did not >> have this issue. It had the same settings (including "Discard old >> builds"), but when I was running jobs through this environment on a regular >> basis, Jenkins and plugins would have been at older versions. >> >> >> >> To hone in on git plugin versions just in the change they are relevant, I >> compared my build.xml: >> >> >> >> New Environment >> >> <hudson.plugins.git.util.BuildData plugin="[email protected]"> >> >> <marked plugin="[email protected]"> >> >> <revision plugin="[email protected]"> >> >> >> >> Old Environment >> >> <hudson.plugins.git.util.BuildData plugin="[email protected]"> >> >> <marked plugin="[email protected]"> >> >> <revision plugin="[email protected]"> >> >> >> >> --------------- >> >> >> >> Given this additional information, please let me know if you have any new >> thoughts or if you feel I should file a bug report (for perhaps the >> logRotate or BuildDiscarder area?) as I continue to research and test. >> >> >> >> Thank you again for your help and everything you do for the Jenkins >> community! >> >> >> >> -Eric >> >> >> >> >> >> >> >> On Fri, Jan 3, 2020 at 8:38 PM Mark Waite <[email protected]> >> wrote: >> >> >> >> >> >> On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson <[email protected]> wrote: >> >> I am troubleshooting a scenario where jobs with a wildcard tag refspec >> (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and >> rebuild a successfully built tag referencing the same commit. >> >> >> >> When a rerun occurs, prior job runs for the tag are no longer present in >> the Git Build History for the job. >> >> >> >> >> >> That sounds to me as though you are relying on the JENKINS-19022 memory >> bloat bug to retain history of more jobs in the BuildData than are actually >> in the build history. >> >> >> >> If someone executed one of the scripts that is mentioned in JENKINS-19022 >> to clean the excess BuildData from the history, that seems like it might >> cause the SHA-1's referenced by some of the tags to be removed from the >> history. >> >> >> >> If build history were being removed based on a specific number of builds, >> then it also could be that the history entry which included that tag was >> removed from the list, and thus was no longer visible. >> >> >> >> Mark Waite >> >> >> >> I am seeking help to further investigate and trace polling activity to >> try to determine what changes are being detected that trigger the job to >> run again, or learn if i should submit this as a bug report for assistance >> instead. >> >> >> >> Thanks, everyone! >> >> >> >> >> >> --------------- >> >> Existing Bugs Found: >> >> >> >> This was the most relative bug I could find, yet my situation is >> different. >> >> >> >> >> https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true >> >> --------------- >> >> >> >> >> >> *My Environment:* >> >> >> >> Jenkins 2.176.3 on traditional VMs (not containers) >> >> GitHub Plugin 1.29.5 >> >> Git Plugin 4.0.0 >> >> >> >> >> >> *Steps to reproduce:* >> >> >> >> 1. Add a webhook to a GitHub repo which sends the following events to >> Jenkins: Pushes, Branch or tag creation, Releases. >> >> >> >> 2. Create two “Pipeline” jobs in Jenkins, both referencing the same >> GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. >> >> >> >> *NOTE: JJB is used to create the jobs (jobs are not copied).* >> >> >> >> Settings for job 1 (master branch job): >> >> Do not allow concurrent builds >> >> GitHub hook trigger for GITScm polling >> >> Pipeline script from SCM: Git >> >> Repo ID: origin >> >> Refspec: +refs/heads/*:refs/remotes/origin/* >> >> Branch Specifier: refs/heads/master >> >> Additional Behaviors: Wipe out repository & force clone >> >> Lightweight checkout: false >> >> >> >> Settings for job 2 (tag job): >> >> Do not allow concurrent builds >> >> GitHub hook trigger for GITScm polling >> >> Pipeline script from SCM: Git >> >> Repo ID: origin >> >> Refspec: +refs/tags/*:refs/remotes/origin/tags/* >> >> Branch Specifier: */tags/* >> >> Additional Behaviors: Wipe out repository & force clone >> >> Lightweight checkout: false >> >> >> >> 3. When commits are merged to the master branch of the repo, the webhook >> fires and Job 1 executes. >> >> >> >> 4. When new releases (tags) are published, the webhook fires and Job 2 >> executes for the most recent (new) tag. >> >> >> >> (Things have been running smoothly for me with this type of setup for a >> couple years.) >> >> >> >> 5. Occasionally there is a situation with jobs configured like job 2, >> specifically: >> >> - New commits are pushed to master which triggers the webhook to >> Jenkins. >> - Job 1 polls and detects the new commits to master and runs. >> - Job 2 polls and detects changes even though no tags have been >> created and previously built tags still reference the same commit. >> - Job 2 then runs and rebuilds the last tag it had already built >> successfully. >> - The polling log shows last built revision is the same commit and >> tag, no diff is performed, but changes are found. >> - The changes page for the job shows no changes. >> >> >> >> Observations when this occurs: >> >> - Let's say the tag has been built 4 times. We will have a polling >> log and change logs for the first job run that initially built the tag, >> then jobs 2 and 3 will have no polling log on the filesystem but will have >> an empty changelog0.xml, then the last build of the tag has a polling log >> and change log files. >> - When a tag is being re-built, when you look at Git Build Data for >> the job rebuilding that tag, all prior job runs that built that tag >> successfully are missing from the history. >> - The problem does not always happen in my environment. It can >> happen for a repo one day then not happen the next. Perhaps I am simply >> overlooking some activity in the repo that results in polling detecting >> changes with my wildcard tag specifier. >> - Workspaces appear to be intact as required for the git plugin to >> perform a wildcard tag poll (post-clone). There are no indications of >> workspaces needing to be created as job runs start. >> - The same build node (same workspace) can be used for all job runs >> and the issue can still occur. >> >> >> >> >> >> *Example polling log and Git Build Data for a re-run:* >> >> >> >> Started on Dec 30, 2019 7:54:51 AM >> >> Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on Mon >> Dec 30 07:54:51 EST 2019 >> >> Using strategy: Default >> >> [poll] Last Built Revision: Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> >> using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913 >> >> > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10 >> >> Fetching changes from the remote Git repositories >> >> > /data/git-client/bin/git config remote.origin.url >> https://github.com/repo.git # timeout=10 >> >> Fetching upstream changes from https://github.com/repo.git >> >> > /data/git-client/bin/git --version # timeout=10 >> >> using GIT_ASKPASS to set credentials xxxx Service Account >> >> > /data/git-client/bin/git fetch --tags --force --progress -- >> https://github.com/repo.git +refs/tags/*:refs/remotes/origin/tags/* # >> timeout=10 >> >> Polling for changes in >> >> Seen branch in repository origin/2.2.0 >> >> Seen branch in repository origin/2.3.0 >> >> Seen branch in repository origin/2.4.0 >> >> Seen branch in repository origin/gh-pages >> >> Seen branch in repository origin/master >> >> Seen branch in repository origin/openshift-migration >> >> Seen branch in repository origin/rhel-build >> >> Seen branch in repository origin/tags/v0.10.0 >> >> Seen branch in repository origin/tags/v0.3 >> >> Seen branch in repository origin/tags/v0.4 >> >> Seen branch in repository origin/tags/v0.7.0 >> >> Seen branch in repository origin/tags/v0.7.1 >> >> Seen branch in repository origin/tags/v0.8.0 >> >> Seen branch in repository origin/tags/v0.9.0 >> >> Seen branch in repository origin/tags/v0.9.1 >> >> Seen branch in repository origin/tags/v1.0.0 >> >> Seen branch in repository origin/tags/v1.0.0-secure >> >> Seen branch in repository origin/tags/v1.1.0 >> >> Seen branch in repository origin/tags/v1.2.0 >> >> Seen branch in repository origin/tags/v1.3.0 >> >> Seen branch in repository origin/tags/v1.3.1 >> >> Seen branch in repository origin/tags/v1.4.0 >> >> Seen branch in repository origin/tags/v1.5.0 >> >> Seen branch in repository origin/tags/v1.6.0 >> >> Seen branch in repository origin/tags/v1.6.1 >> >> Seen branch in repository origin/tags/v1.7.0 >> >> Seen branch in repository origin/tags/v1.7.1 >> >> Seen branch in repository origin/tags/v1.7.2 >> >> Seen branch in repository origin/tags/v1.8.0 >> >> Seen branch in repository origin/tags/v1.9.0 >> >> Seen branch in repository origin/tags/v2.0.0 >> >> Seen branch in repository origin/tags/v2.1.0 >> >> Seen branch in repository origin/tags/v2.2.0 >> >> Seen branch in repository origin/tags/v2.3.0 >> >> Seen 34 remote branches >> >> > /data/git-client/bin/git show-ref --tags -d # timeout=10 >> >> Using strategy: Default >> >> [poll] Last Built Revision: Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> >> Done. Took 0.53 sec >> >> Changes found >> >> >> >> --------------------------------- >> >> >> >> Git Build Data (for Build #9) >> >> Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc >> >> origin/tags/v2.4.0 >> >> refs/tags/v2.4.0 >> >> Built Branches >> >> refs/tags/v2.1.0: Build #4 of Revision >> 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, >> refs/tags/v2.1.0) >> >> origin/tags/v2.2.0: Build #5 of Revision >> 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, >> refs/tags/v2.2.0) >> >> refs/tags/v2.2.0: Build #5 of Revision >> 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, >> refs/tags/v2.2.0) >> >> origin/tags/v2.1.0: Build #4 of Revision >> 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, >> refs/tags/v2.1.0) >> >> origin/tags/v2.0.0: Build #2 of Revision >> 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, >> refs/tags/v2.0.0) >> >> refs/tags/v2.4.0: Build #9 of Revision >> 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, >> refs/tags/v2.4.0) >> >> refs/tags/v2.3.0: Build #8 of Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> >> origin/tags/v1.0.0-secure: Build #1 of Revision >> 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, >> refs/tags/v1.0.0-secure) >> >> origin/tags/v2.4.0: Build #9 of Revision >> 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, >> refs/tags/v2.4.0) >> >> refs/tags/v1.0.0-secure: Build #1 of Revision >> 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, >> refs/tags/v1.0.0-secure) >> >> refs/tags/v2.0.0: Build #2 of Revision >> 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, >> refs/tags/v2.0.0) >> >> origin/tags/v2.3.0: Build #8 of Revision >> f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, >> refs/tags/v2.3.0) >> >> >> >> NOTE: >> >> Build #6 was the initial build of refs/tags/v2.3.0 >> >> Build #7 was the second rerun of refs/tags/v2.3.0 >> >> Build #8 was the last rerun of refs/tags/v2.3.0 >> >> Build #9 was for refs/tags/v2.4.0 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com >> <https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> >> >> -- >> >> Thanks! >> >> Mark Waite >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Jenkins Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jenkinsci-users/_JJM5DZoPvA/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> >> >> >> -- >> >> Thanks! >> >> Mark Waite >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGNKPNDDfBNKwbN22%2BVK18VFSmT6%2BWYD1syv95Ucs98vg%40mail.gmail.com >> <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGNKPNDDfBNKwbN22%2BVK18VFSmT6%2BWYD1syv95Ucs98vg%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Jenkins Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jenkinsci-users/_JJM5DZoPvA/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/AM0PR03MB571566E46420E5508F3069F7CF390%40AM0PR03MB5715.eurprd03.prod.outlook.com >> <https://groups.google.com/d/msgid/jenkinsci-users/AM0PR03MB571566E46420E5508F3069F7CF390%40AM0PR03MB5715.eurprd03.prod.outlook.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxn17p1QJ-3RAqHJCqEHGqt1ka_YqAUd6oKii1%3DJ5S2zFQ%40mail.gmail.com.
