I am troubleshooting a scenario where jobs with a wildcard tag refspec (+refs/tags/*:refs/remotes/origin/tags/*) sometimes detect changes and rebuild a successfully built tag referencing the same commit.
When a rerun occurs, prior job runs for the tag are no longer present in the Git Build History for the job. I am seeking help to further investigate and trace polling activity to try to determine what changes are being detected that trigger the job to run again, or learn if i should submit this as a bug report for assistance instead. Thanks, everyone! --------------- Existing Bugs Found: This was the most relative bug I could find, yet my situation is different. https://issues.jenkins-ci.org/browse/JENKINS-17614?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true --------------- *My Environment:* Jenkins 2.176.3 on traditional VMs (not containers) GitHub Plugin 1.29.5 Git Plugin 4.0.0 *Steps to reproduce:* 1. Add a webhook to a GitHub repo which sends the following events to Jenkins: Pushes, Branch or tag creation, Releases. 2. Create two “Pipeline” jobs in Jenkins, both referencing the same GitHub repo. The jobs execute a Declarative Pipeline Jenkinsfile script. *NOTE: JJB is used to create the jobs (jobs are not copied).* Settings for job 1 (master branch job): Do not allow concurrent builds GitHub hook trigger for GITScm polling Pipeline script from SCM: Git Repo ID: origin Refspec: +refs/heads/*:refs/remotes/origin/* Branch Specifier: refs/heads/master Additional Behaviors: Wipe out repository & force clone Lightweight checkout: false Settings for job 2 (tag job): Do not allow concurrent builds GitHub hook trigger for GITScm polling Pipeline script from SCM: Git Repo ID: origin Refspec: +refs/tags/*:refs/remotes/origin/tags/* Branch Specifier: */tags/* Additional Behaviors: Wipe out repository & force clone Lightweight checkout: false 3. When commits are merged to the master branch of the repo, the webhook fires and Job 1 executes. 4. When new releases (tags) are published, the webhook fires and Job 2 executes for the most recent (new) tag. (Things have been running smoothly for me with this type of setup for a couple years.) 5. Occasionally there is a situation with jobs configured like job 2, specifically: - New commits are pushed to master which triggers the webhook to Jenkins. - Job 1 polls and detects the new commits to master and runs. - Job 2 polls and detects changes even though no tags have been created and previously built tags still reference the same commit. - Job 2 then runs and rebuilds the last tag it had already built successfully. - The polling log shows last built revision is the same commit and tag, no diff is performed, but changes are found. - The changes page for the job shows no changes. Observations when this occurs: - Let's say the tag has been built 4 times. We will have a polling log and change logs for the first job run that initially built the tag, then jobs 2 and 3 will have no polling log on the filesystem but will have an empty changelog0.xml, then the last build of the tag has a polling log and change log files. - When a tag is being re-built, when you look at Git Build Data for the job rebuilding that tag, all prior job runs that built that tag successfully are missing from the history. - The problem does not always happen in my environment. It can happen for a repo one day then not happen the next. Perhaps I am simply overlooking some activity in the repo that results in polling detecting changes with my wildcard tag specifier. - Workspaces appear to be intact as required for the git plugin to perform a wildcard tag poll (post-clone). There are no indications of workspaces needing to be created as job runs start. - The same build node (same workspace) can be used for all job runs and the issue can still occur. *Example polling log and Git Build Data for a re-run:* Started on Dec 30, 2019 7:54:51 AM Started by event from 10.25.59.190 ⇒ https://xxxx/github-webhook/ on Mon Dec 30 07:54:51 EST 2019 Using strategy: Default [poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) using credential 28d104ae-ad72-401a-8dc2-d72cf4b8e913 > /data/git-client/bin/git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories > /data/git-client/bin/git config remote.origin.url https://github.com/repo.git # timeout=10 Fetching upstream changes from https://github.com/repo.git > /data/git-client/bin/git --version # timeout=10 using GIT_ASKPASS to set credentials xxxx Service Account > /data/git-client/bin/git fetch --tags --force --progress -- https://github.com/repo.git +refs/tags/*:refs/remotes/origin/tags/* # timeout=10 Polling for changes in Seen branch in repository origin/2.2.0 Seen branch in repository origin/2.3.0 Seen branch in repository origin/2.4.0 Seen branch in repository origin/gh-pages Seen branch in repository origin/master Seen branch in repository origin/openshift-migration Seen branch in repository origin/rhel-build Seen branch in repository origin/tags/v0.10.0 Seen branch in repository origin/tags/v0.3 Seen branch in repository origin/tags/v0.4 Seen branch in repository origin/tags/v0.7.0 Seen branch in repository origin/tags/v0.7.1 Seen branch in repository origin/tags/v0.8.0 Seen branch in repository origin/tags/v0.9.0 Seen branch in repository origin/tags/v0.9.1 Seen branch in repository origin/tags/v1.0.0 Seen branch in repository origin/tags/v1.0.0-secure Seen branch in repository origin/tags/v1.1.0 Seen branch in repository origin/tags/v1.2.0 Seen branch in repository origin/tags/v1.3.0 Seen branch in repository origin/tags/v1.3.1 Seen branch in repository origin/tags/v1.4.0 Seen branch in repository origin/tags/v1.5.0 Seen branch in repository origin/tags/v1.6.0 Seen branch in repository origin/tags/v1.6.1 Seen branch in repository origin/tags/v1.7.0 Seen branch in repository origin/tags/v1.7.1 Seen branch in repository origin/tags/v1.7.2 Seen branch in repository origin/tags/v1.8.0 Seen branch in repository origin/tags/v1.9.0 Seen branch in repository origin/tags/v2.0.0 Seen branch in repository origin/tags/v2.1.0 Seen branch in repository origin/tags/v2.2.0 Seen branch in repository origin/tags/v2.3.0 Seen 34 remote branches > /data/git-client/bin/git show-ref --tags -d # timeout=10 Using strategy: Default [poll] Last Built Revision: Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) Done. Took 0.53 sec Changes found --------------------------------- Git Build Data (for Build #9) Revision: 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc origin/tags/v2.4.0 refs/tags/v2.4.0 Built Branches refs/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0) origin/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0) refs/tags/v2.2.0: Build #5 of Revision 2f0b66b9cf8b6f9852a461c31f435196c9270f71 (origin/tags/v2.2.0, refs/tags/v2.2.0) origin/tags/v2.1.0: Build #4 of Revision 42d67f58dc99a77c91eca90132a065ebca4f5c66 (origin/tags/v2.1.0, refs/tags/v2.1.0) origin/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0) refs/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0) refs/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) origin/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure) origin/tags/v2.4.0: Build #9 of Revision 702b6f79d9f302d04e0647da83afc5f2b7ef8ebc (origin/tags/v2.4.0, refs/tags/v2.4.0) refs/tags/v1.0.0-secure: Build #1 of Revision 3e54ce51fbc67ea8f3f59d3a2389fe0aca2ba15d (origin/tags/v1.0.0-secure, refs/tags/v1.0.0-secure) refs/tags/v2.0.0: Build #2 of Revision 11f93f5e9eb5fa87745f4db062b6ee48f4975350 (origin/tags/v2.0.0, refs/tags/v2.0.0) origin/tags/v2.3.0: Build #8 of Revision f44c2c56f44b84a5d2b534eacf9c51a099f65dc2 (origin/tags/v2.3.0, refs/tags/v2.3.0) NOTE: Build #6 was the initial build of refs/tags/v2.3.0 Build #7 was the second rerun of refs/tags/v2.3.0 Build #8 was the last rerun of refs/tags/v2.3.0 Build #9 was for refs/tags/v2.4.0 -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b114b34b-3f60-4dc5-87cd-072bcfb81f8c%40googlegroups.com.
