Yes, I intend to write up the issue today for tracking. I have been troubleshooting more but unfortunately don't have anything valuable to add just yet. I will reply to this thread once the issue has been opened with more details.
Eric On Thu, Jan 9, 2020, 05:49 Beushausen, Christian < [email protected]> wrote: > Hi Eric, Mark, > > > > Could you keep the list updated? Or have you created a new issue on Jira > for this? > We do see this (or a very similar) issue also for at least one job in our > environment > > > > Thank you and cheers. > > > > Mit freundlichen Gruessen/Best regards, > > *Christian Beushausen* > I S&T PD SW SWF > Interior Systems & Technology > > *E-Mail:* [email protected] > > > *From:* [email protected] <[email protected]> > *On Behalf Of *Mark Waite > *Sent:* Dienstag, 7. Januar 2020 01:21 > *To:* Jenkins Users <[email protected]> > *Subject:* Re: Jobs with a wildcard tag refspec sometimes rebuild tags at > same commit > > > > Discard old builds is very likely the problem. If a tag build is removed > from the history by "Discard old builds" then future searches for the that > tag (or the SHA-1 of the commit associated with the tag) will fail and it > will build again. > > > > On Mon, Jan 6, 2020 at 5:18 PM Eric Pierson <[email protected]> wrote: > > Thank you for your response, Mark! > > > > I have not tried to run any scripts to prune history as I see described in > JENKINS-19022. There are a lot of comments in this one and I am still > re-reading them to be sure I am not missing something else relevant folks > discovered along the way. > > > > I do have "Discard old builds" enabled with Strategy "Log Rotation" and > "Max # of builds to keep" = 10. Forgive me for omitting the fact that I > use Discard old builds initially. > > > > Reading more about "Discard old builds": > > - Build count: discard the *oldest build* when a certain number of > builds already exist > - Note that Jenkins does not discard items immediately when this > configuration is updated, or as soon as any of the configured values are > exceeded; *these rules are evaluated each time a build of this project > completes.* > > I am now focusing on the "Discard old builds" settings and build.xml > content, and what may be happening to the build history when a build > completes. I have one job that has only run a total of 7 times that had > the issue on builds #3 and #4, and the example I initially provided has > only run a total of 12 times and we see the issue on multiple builds there, > too. So while there are indeed many tags in the repo, the build history is > not "large" yet. > > > > I have another Jenkins environment I used for a while where I did not have > this issue. It had the same settings (including "Discard old builds"), but > when I was running jobs through this environment on a regular basis, > Jenkins and plugins would have been at older versions. > > > > To hone in on git plugin versions just in the change they are relevant, I > compared my build.xml: > > > > New Environment > > <hudson.plugins.git.util.BuildData plugin="[email protected]"> > > <marked plugin="[email protected]"> > > <revision plugin="[email protected]"> > > > > Old Environment > > <hudson.plugins.git.util.BuildData plugin="[email protected]"> > > <marked plugin="[email protected]"> > > <revision plugin="[email protected]"> > > > > --------------- > > > > Given this additional information, please let me know if you have any new > thoughts or if you feel I should file a bug report (for perhaps the > logRotate or BuildDiscarder area?) as I continue to research and test. > > > > Thank you again for your help and everything you do for the Jenkins > community! > > > > -Eric > > > > > > > > On Fri, Jan 3, 2020 at 8:38 PM Mark Waite <[email protected]> > wrote: > > > > > > On Fri, Jan 3, 2020 at 2:51 PM Eric Pierson <[email protected]> wrote: > > I am troubleshooting a scenario where jobs with a wildcard tag refspec > (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and > rebuild a successfully built tag referencing the same commit. > > > > When a rerun occurs, prior job runs for the tag are no longer present in > the Git Build History for the job. > > > > > > That sounds to me as though you are relying on the JENKINS-19022 memory > bloat bug to retain history of more jobs in the BuildData than are actually > in the build history. > > > > If someone executed one of the scripts that is mentioned in JENKINS-19022 > to clean the excess BuildData from the history, that seems like it might > cause the SHA-1's referenced by some of the tags to be removed from the > history. > > > > If build history were being removed based on a specific number of builds, > then it also could be that the history entry which included that tag was > removed from the list, and thus was no longer visible. > > > > Mark Waite > > > > I am seeking help to further investigate and trace polling activity to try > to determine what changes are being detected that trigger the job to run > again, or learn if i should submit this as a bug report for assistance > instead. > > > > Thanks, everyone! > > > > > > --------------- > > Existing Bugs Found: > > > > This was the most relative bug I could find, yet my situation is different. > > > > > https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true > > --------------- > > > > > > *My Environment:* > > > > Jenkins 2.176.3 on traditional VMs (not containers) > > GitHub Plugin 1.29.5 > > Git Plugin 4.0.0 > > > > > > *Steps to reproduce:* > > > > 1. Add a webhook to a GitHub repo which sends the following events to > Jenkins: Pushes, Branch or tag creation, Releases. > > > > 2. Create two “Pipeline” jobs in Jenkins, both referencing the same GitHub > repo. The jobs execute a Declarative Pipeline Jenkinsfile script. > > > > *NOTE: JJB is used to create the jobs (jobs are not copied).* > > > > Settings for job 1 (master branch job): > > Do not allow concurrent builds > > GitHub hook trigger for GITScm polling > > Pipeline script from SCM: Git > > Repo ID: origin > > Refspec: +refs/heads/*:refs/remotes/origin/* > > Branch Specifier: refs/heads/master > > Additional Behaviors: Wipe out repository & force clone > > Lightweight checkout: false > > > > Settings for job 2 (tag job): > > Do not allow concurrent builds > > GitHub hook trigger for GITScm polling > > Pipeline script from SCM: Git > > Repo ID: origin > > Refspec: +refs/tags/*:refs/remotes/origin/tags/* > > Branch Specifier: */tags/* > > Additional Behaviors: Wipe out repository & force clone > > Lightweight checkout: false > > > > 3. When commits are merged to the master branch of the repo, the webhook > fires and Job 1 executes. > > > > 4. When new releases (tags) are published, the webhook fires and Job 2 > executes for the most recent (new) tag. > > > > (Things have been running smoothly for me with this type of setup for a > couple years.) > > > > 5. Occasionally there is a situation with jobs configured like job 2, > specifically: > > - New commits are pushed to master which triggers the webhook to > Jenkins. > - Job 1 polls and detects the new commits to master and runs. > - Job 2 polls and detects changes even though no tags have been > created and previously built tags still reference the same commit. > - Job 2 then runs and rebuilds the last tag it had already built > successfully. > - The polling log shows last built revision is the same commit and > tag, no diff is performed, but changes are found. > - The changes page for the job shows no changes. > > > > Observations when this occurs: > > - Let's say the tag has been built 4 times. We will have a polling > log and change logs for the first job run that initially built the tag, > then jobs 2 and 3 will have no polling log on the filesystem but will have > an empty changelog0.xml, then the last build of the tag has a polling log > and change log files. > - When a tag is being re-built, when you look at Git Build Data for > the job rebuilding that tag, all prior job runs that built that tag > successfully are missing from the history. > - The problem does not always happen in my environment. It can happen > for a repo one day then not happen the next. Perhaps I am simply > overlooking some activity in the repo that results in polling detecting > changes with my wildcard tag specifier. > - Workspaces appear to be intact as required for the git plugin to > perform a wildcard tag poll (post-clone). There are no indications of > workspaces needing to be created as job runs start. > - The same build node (same workspace) can be used for all job runs > and the issue can still occur. > > > > > > *Example polling log and Git Build Data for a re-run:* > > > > Started on Dec 30, 2019 7:54:51 AM > > Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on Mon > Dec 30 07:54:51 EST 2019 > > Using strategy: Default > > [poll] Last Built Revision: Revision > f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, > refs/tags/v2.3.0) > > using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913 > > > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10 > > Fetching changes from the remote Git repositories > > > /data/git-client/bin/git config remote.origin.url > https://github.com/repo.git # timeout=10 > > Fetching upstream changes from https://github.com/repo.git > > > /data/git-client/bin/git --version # timeout=10 > > using GIT_ASKPASS to set credentials xxxx Service Account > > > /data/git-client/bin/git fetch --tags --force --progress -- > https://github.com/repo.git +refs/tags/*:refs/remotes/origin/tags/* # > timeout=10 > > Polling for changes in > > Seen branch in repository origin/2.2.0 > > Seen branch in repository origin/2.3.0 > > Seen branch in repository origin/2.4.0 > > Seen branch in repository origin/gh-pages > > Seen branch in repository origin/master > > Seen branch in repository origin/openshift-migration > > Seen branch in repository origin/rhel-build > > Seen branch in repository origin/tags/v0.10.0 > > Seen branch in repository origin/tags/v0.3 > > Seen branch in repository origin/tags/v0.4 > > Seen branch in repository origin/tags/v0.7.0 > > Seen branch in repository origin/tags/v0.7.1 > > Seen branch in repository origin/tags/v0.8.0 > > Seen branch in repository origin/tags/v0.9.0 > > Seen branch in repository origin/tags/v0.9.1 > > Seen branch in repository origin/tags/v1.0.0 > > Seen branch in repository origin/tags/v1.0.0-secure > > Seen branch in repository origin/tags/v1.1.0 > > Seen branch in repository origin/tags/v1.2.0 > > Seen branch in repository origin/tags/v1.3.0 > > Seen branch in repository origin/tags/v1.3.1 > > Seen branch in repository origin/tags/v1.4.0 > > Seen branch in repository origin/tags/v1.5.0 > > Seen branch in repository origin/tags/v1.6.0 > > Seen branch in repository origin/tags/v1.6.1 > > Seen branch in repository origin/tags/v1.7.0 > > Seen branch in repository origin/tags/v1.7.1 > > Seen branch in repository origin/tags/v1.7.2 > > Seen branch in repository origin/tags/v1.8.0 > > Seen branch in repository origin/tags/v1.9.0 > > Seen branch in repository origin/tags/v2.0.0 > > Seen branch in repository origin/tags/v2.1.0 > > Seen branch in repository origin/tags/v2.2.0 > > Seen branch in repository origin/tags/v2.3.0 > > Seen 34 remote branches > > > /data/git-client/bin/git show-ref --tags -d # timeout=10 > > Using strategy: Default > > [poll] Last Built Revision: Revision > f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, > refs/tags/v2.3.0) > > Done. Took 0.53 sec > > Changes found > > > > --------------------------------- > > > > Git Build Data (for Build #9) > > Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc > > origin/tags/v2.4.0 > > refs/tags/v2.4.0 > > Built Branches > > refs/tags/v2.1.0: Build #4 of Revision > 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, > refs/tags/v2.1.0) > > origin/tags/v2.2.0: Build #5 of Revision > 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, > refs/tags/v2.2.0) > > refs/tags/v2.2.0: Build #5 of Revision > 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, > refs/tags/v2.2.0) > > origin/tags/v2.1.0: Build #4 of Revision > 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, > refs/tags/v2.1.0) > > origin/tags/v2.0.0: Build #2 of Revision > 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, > refs/tags/v2.0.0) > > refs/tags/v2.4.0: Build #9 of Revision > 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, > refs/tags/v2.4.0) > > refs/tags/v2.3.0: Build #8 of Revision > f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, > refs/tags/v2.3.0) > > origin/tags/v1.0.0-secure: Build #1 of Revision > 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, > refs/tags/v1.0.0-secure) > > origin/tags/v2.4.0: Build #9 of Revision > 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, > refs/tags/v2.4.0) > > refs/tags/v1.0.0-secure: Build #1 of Revision > 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, > refs/tags/v1.0.0-secure) > > refs/tags/v2.0.0: Build #2 of Revision > 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, > refs/tags/v2.0.0) > > origin/tags/v2.3.0: Build #8 of Revision > f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, > refs/tags/v2.3.0) > > > > NOTE: > > Build #6 was the initial build of refs/tags/v2.3.0 > > Build #7 was the second rerun of refs/tags/v2.3.0 > > Build #8 was the last rerun of refs/tags/v2.3.0 > > Build #9 was for refs/tags/v2.4.0 > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com > <https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > > > -- > > Thanks! > > Mark Waite > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-users/_JJM5DZoPvA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGFkmt76_HEU4h_Aey%2Bai8UiD3hYkwNE%3Da3ZaF70SK-JA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vxkMvHNW%2B8JpOUPFj%2BqRHgf%2Bi-6%3DR-R0Ppmam4XjDPXO-A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > > > -- > > Thanks! > > Mark Waite > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGNKPNDDfBNKwbN22%2BVK18VFSmT6%2BWYD1syv95Ucs98vg%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-users/CAO49JtGNKPNDDfBNKwbN22%2BVK18VFSmT6%2BWYD1syv95Ucs98vg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Jenkins Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/jenkinsci-users/_JJM5DZoPvA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-users/AM0PR03MB571566E46420E5508F3069F7CF390%40AM0PR03MB5715.eurprd03.prod.outlook.com > <https://groups.google.com/d/msgid/jenkinsci-users/AM0PR03MB571566E46420E5508F3069F7CF390%40AM0PR03MB5715.eurprd03.prod.outlook.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAKh5vx%3DAUMWabH2PS39TOPv%3DqR-oP0d2oyK6%3D8FSbnPtqboRAQ%40mail.gmail.com.
