Hi Mark - whilst looking for a place to stash my reference repo I discovered that Jenkins is keeping a cache of git checkouts in the /var/lib/jenkins/cache folder, and by deleting these files the multibranch pipeline scan now succeeds, so there was obviously something corrupt in that cache (possibly caused by a recent instability issue with the Jenkins server hardware), so I've not implemented a reference repo. Thanks for your help ;-)
Kevin On Tuesday, 2 July 2019 15:25:41 UTC+1, Kevin Stevens wrote: > > Thanks for the responses and the issue link Mark, I've increased the Git > timeout value as described, and confirmed in the log that the 30 minute > timeout is in effect, but I'm still seeing the same failure (i.e. fatal > error in Scan Multibranch Pipeline Now). > I was pretty confident that there's not a timeout issue as the failure > occurs in a matter of seconds, so I'm going to try the reference repo > approach which seems like it should work, but it would appear to be an > issue within the git client parsing to me. I'll confirm my outcome. > > Regards, Kevin > > On Tuesday, 2 July 2019 13:37:41 UTC+1, Mark Waite wrote: >> >> Sorry, I didn't read your description thoroughly enough. You said: >> >> I'm getting a fatal error reported when doing a repository scan (Scan >>> Multibranch Pipeline Now) which was previously working. >>> >> >> The command line git log output indicates that your repository is large >> enough that it may be reaching the default 10 minute clone timeout. Refer >> to https://issues.jenkins-ci.org/browse/JENKINS-38973 for a discussion >> of possible workarounds. >> >> On Tue, Jul 2, 2019 at 6:31 AM Mark Waite <[email protected]> wrote: >> >>> The repository in the workspace on the agent running that build is >>> probably damaged. Wipe the workspace on the agent and run the job again. >>> >>> If wiping the workspace and running the job again does not resolve it, >>> you may also be encountering a timeout while cloning the repository. >>> Increase the timeout for the repository clone from its default of 10 >>> minutes to something large enough that it will allow you to clone the >>> repository. >>> >>> The amount of output you're showing hints that the repository is large >>> or the network connection between the agent and the upstream cloned >>> repository is slow. In either of those cases, you probably want to reduce >>> the clone time and the disc space use by applying one or more of the >>> techniques described in "Git in the Large >>> <https://youtu.be/jBGFjFc6Jf8?t=6434>". Those techniques include (1) >>> reference repositories on the agent, (2) narrow refspecs, (3) shallow >>> clones, and (4) sparse checkouts. >>> >>> If this is a Pipeline repository and the failure is during the initial >>> clone of the repository, then you may also need to use lightweight checkout >>> to only checkout the Jenkinsfile rather than the entire repository. >>> Alternately for Pipelines, if your git provider is GitHub, Bitbucket, or >>> Gitea, you can significantly improve performance by using those branch >>> source plugins to manage the Pipeline instead of relying on low-level >>> command line git calls. >>> >>> Reference repositories are usually the most effective technique to >>> reduce data transfer time and disc space use for large git repositories. >>> Allan Burdajewicz of CloudBees wrote a great article on reference >>> repositories at >>> https://support.cloudbees.com/hc/en-us/articles/115001728812-Using-a-Git-reference-repository >>> . >>> >>> On Tue, Jul 2, 2019 at 5:51 AM Kevin Stevens <[email protected]> wrote: >>> >>>> I'm getting a fatal error reported when doing a repository scan (Scan >>>> Multibranch Pipeline Now) which was previously working. >>>> I've run the git commands manually from the command line (copied and >>>> pasted) and they appear to work correctly, so it seems like a problem with >>>> the Jenkins git client plugin. >>>> >>>> The scan log is below (repository name and account information modified >>>> for security reasons). >>>> I'm not clear if this is a GIT or a Jenkins git client problem. I've >>>> cleared the Jenkins workspace and updated to the latest Jenkins plugins >>>> (running on Ubuntu 18.04). >>>> Is anyone able to offer help in debugging this issue please? >>>> >>>> >>>> [Tue Jul 02 09:48:00 BST 2019] Starting branch indexing... >>>> > git --version # timeout=10 >>>> using GIT_ASKPASS to set credentials Jenkins Bitbucket User >>>> > git ls-remote --symref >>>> [email protected]:my-company-name/my-repository-name.git # timeout=10 >>>> > git rev-parse --is-inside-work-tree # timeout=10 >>>> Setting origin to >>>> [email protected]:my-company-name/my-repository-name.git >>>> > git config remote.origin.url >>>> [email protected]:my-company-name/my-repository-name.git # timeout=10 >>>> Fetching & pruning origin... >>>> Listing remote references... >>>> > git config --get remote.origin.url # timeout=10 >>>> > git --version # timeout=10 >>>> using GIT_ASKPASS to set credentials Jenkins Bitbucket User >>>> > git ls-remote -h >>>> [email protected]:my-company-name/my-repository-name.git # timeout=10 >>>> Fetching upstream changes from origin >>>> > git config --get remote.origin.url # timeout=10 >>>> using GIT_ASKPASS to set credentials Jenkins Bitbucket User >>>> > git fetch --tags --progress origin >>>> +refs/heads/*:refs/remotes/origin/* --prune >>>> ERROR: [Tue Jul 02 09:48:10 BST 2019] Could not fetch branches from >>>> source 498872a6-3888-419f-9d2f-a2baa4520968 >>>> hudson.plugins.git.GitException: Command "git fetch --tags --progress >>>> origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code >>>> 128: >>>> stdout: >>>> stderr: remote: Counting objects: 114, done. >>>> remote: Compressing objects: 0% (1/114) >>>> remote: Compressing objects: 1% (2/114) >>>> remote: Compressing objects: 2% (3/114) >>>> remote: Compressing objects: 3% (4/114) >>>> remote: Compressing objects: 4% (5/114) >>>> remote: Compressing objects: 5% (6/114) >>>> remote: Compressing objects: 6% (7/114) >>>> remote: Compressing objects: 7% (8/114) >>>> remote: Compressing objects: 8% (10/114) >>>> remote: Compressing objects: 9% (11/114) >>>> remote: Compressing objects: 10% (12/114) >>>> remote: Compressing objects: 11% (13/114) >>>> remote: Compressing objects: 12% (14/114) >>>> remote: Compressing objects: 13% (15/114) >>>> remote: Compressing objects: 14% (16/114) >>>> remote: Compressing objects: 15% (18/114) >>>> remote: Compressing objects: 16% (19/114) >>>> remote: Compressing objects: 17% (20/114) >>>> remote: Compressing objects: 18% (21/114) >>>> remote: Compressing objects: 19% (22/114) >>>> remote: Compressing objects: 20% (23/114) >>>> remote: Compressing objects: 21% (24/114) >>>> remote: Compressing objects: 22% (26/114) >>>> remote: Compressing objects: 23% (27/114) >>>> remote: Compressing objects: 24% (28/114) >>>> remote: Compressing objects: 25% (29/114) >>>> remote: Compressing objects: 26% (30/114) >>>> remote: Compressing objects: 27% (31/114) >>>> remote: Compressing objects: 28% (32/114) >>>> remote: Compressing objects: 29% (34/114) >>>> remote: Compressing objects: 30% (35/114) >>>> remote: Compressing objects: 31% (36/114) >>>> remote: Compressing objects: 32% (37/114) >>>> remote: Compressing objects: 33% (38/114) >>>> remote: Compressing objects: 34% (39/114) >>>> remote: Compressing objects: 35% (40/114) >>>> remote: Compressing objects: 36% (42/114) >>>> remote: Compressing objects: 37% (43/114) >>>> remote: Compressing objects: 38% (44/114) >>>> remote: Compressing objects: 39% (45/114) >>>> remote: Compressing objects: 40% (46/114) >>>> remote: Compressing objects: 41% (47/114) >>>> remote: Compressing objects: 42% (48/114) >>>> remote: Compressing objects: 43% (50/114) >>>> remote: Compressing objects: 44% (51/114) >>>> remote: Compressing objects: 45% (52/114) >>>> remote: Compressing objects: 46% (53/114) >>>> remote: Compressing objects: 47% (54/114) >>>> remote: Compressing objects: 48% (55/114) >>>> remote: Compressing objects: 49% (56/114) >>>> remote: Compressing objects: 50% (57/114) >>>> remote: Compressing objects: 51% (59/114) >>>> remote: Compressing objects: 52% (60/114) >>>> remote: Compressing objects: 53% (61/114) >>>> remote: Compressing objects: 54% (62/114) >>>> remote: Compressing objects: 55% (63/114) >>>> remote: Compressing objects: 56% (64/114) >>>> remote: Compressing objects: 57% (65/114) >>>> remote: Compressing objects: 58% (67/114) >>>> remote: Compressing objects: 59% (68/114) >>>> remote: Compressing objects: 60% (69/114) >>>> remote: Compressing objects: 61% (70/114) >>>> remote: Compressing objects: 62% (71/114) >>>> remote: Compressing objects: 63% (72/114) >>>> remote: Compressing objects: 64% (73/114) >>>> remote: Compressing objects: 65% (75/114) >>>> remote: Compressing objects: 66% (76/114) >>>> remote: Compressing objects: 67% (77/114) >>>> remote: Compressing objects: 68% (78/114) >>>> remote: Compressing objects: 69% (79/114) >>>> remote: Compressing objects: 70% (80/114) >>>> remote: Compressing objects: 71% (81/114) >>>> remote: Compressing objects: 72% (83/114) >>>> remote: Compressing objects: 73% (84/114) >>>> remote: Compressing objects: 74% (85/114) >>>> remote: Compressing objects: 75% (86/114) >>>> remote: Compressing objects: 76% (87/114) >>>> remote: Compressing objects: 77% (88/114) >>>> remote: Compressing objects: 78% (89/114) >>>> remote: Compressing objects: 79% (91/114) >>>> remote: Compressing objects: 80% (92/114) >>>> remote: Compressing objects: 81% (93/114) >>>> remote: Compressing objects: 82% (94/114) >>>> remote: Compressing objects: 83% (95/114) >>>> remote: Compressing objects: 84% (96/114) >>>> remote: Compressing objects: 85% (97/114) >>>> remote: Compressing objects: 86% (99/114) >>>> remote: Compressing objects: 87% (100/114) >>>> remote: Compressing objects: 88% (101/114) >>>> remote: Compressing objects: 89% (102/114) >>>> remote: Compressing objects: 90% (103/114) >>>> remote: Compressing objects: 91% (104/114) >>>> remote: Compressing objects: 92% (105/114) >>>> remote: Compressing objects: 93% (107/114) >>>> remote: Compressing objects: 94% (108/114) >>>> remote: Compressing objects: 95% (109/114) >>>> remote: Compressing objects: 96% (110/114) >>>> remote: Compressing objects: 97% (111/114) >>>> remote: Compressing objects: 98% (112/114) >>>> remote: Compressing objects: 99% (113/114) >>>> remote: Compressing objects: 100% (114/114) >>>> remote: Compressing objects: 100% (114/114), done. >>>> Receiving objects: 0% (1/114) >>>> Receiving objects: 1% (2/114) >>>> Receiving objects: 2% (3/114) >>>> Receiving objects: 3% (4/114) >>>> Receiving objects: 4% (5/114) >>>> Receiving objects: 5% (6/114) >>>> Receiving objects: 6% (7/114) >>>> Receiving objects: 7% (8/114) >>>> Receiving objects: 8% (10/114) >>>> Receiving objects: 9% (11/114) >>>> Receiving objects: 10% (12/114) >>>> Receiving objects: 11% (13/114) >>>> Receiving objects: 12% (14/114) >>>> Receiving objects: 13% (15/114) >>>> Receiving objects: 14% (16/114) >>>> error: object file >>>> .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty >>>> fatal: cannot read existing object info >>>> 21a28795e9c6fa55f026860e0c2f0b08d1b31611 >>>> fatal: index-pack failed >>>> >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556) >>>> at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) >>>> at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) >>>> at >>>> jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634) >>>> at >>>> com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277) >>>> at >>>> com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164) >>>> at >>>> jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025) >>>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>>> at hudson.model.Executor.run(Executor.java:429) >>>> [Tue Jul 02 09:48:10 BST 2019] Finished branch indexing. Indexing took >>>> 10 sec >>>> FATAL: Failed to recompute children of my-repository-name >>>> hudson.plugins.git.GitException: Command "git fetch --tags --progress >>>> origin +refs/heads/*:refs/remotes/origin/* --prune" returned status code >>>> 128: >>>> stdout: >>>> stderr: remote: Counting objects: 114, done. >>>> remote: Compressing objects: 0% (1/114) >>>> remote: Compressing objects: 1% (2/114) >>>> remote: Compressing objects: 2% (3/114) >>>> remote: Compressing objects: 3% (4/114) >>>> remote: Compressing objects: 4% (5/114) >>>> remote: Compressing objects: 5% (6/114) >>>> remote: Compressing objects: 6% (7/114) >>>> remote: Compressing objects: 7% (8/114) >>>> remote: Compressing objects: 8% (10/114) >>>> remote: Compressing objects: 9% (11/114) >>>> remote: Compressing objects: 10% (12/114) >>>> remote: Compressing objects: 11% (13/114) >>>> remote: Compressing objects: 12% (14/114) >>>> remote: Compressing objects: 13% (15/114) >>>> remote: Compressing objects: 14% (16/114) >>>> remote: Compressing objects: 15% (18/114) >>>> remote: Compressing objects: 16% (19/114) >>>> remote: Compressing objects: 17% (20/114) >>>> remote: Compressing objects: 18% (21/114) >>>> remote: Compressing objects: 19% (22/114) >>>> remote: Compressing objects: 20% (23/114) >>>> remote: Compressing objects: 21% (24/114) >>>> remote: Compressing objects: 22% (26/114) >>>> remote: Compressing objects: 23% (27/114) >>>> remote: Compressing objects: 24% (28/114) >>>> remote: Compressing objects: 25% (29/114) >>>> remote: Compressing objects: 26% (30/114) >>>> remote: Compressing objects: 27% (31/114) >>>> remote: Compressing objects: 28% (32/114) >>>> remote: Compressing objects: 29% (34/114) >>>> remote: Compressing objects: 30% (35/114) >>>> remote: Compressing objects: 31% (36/114) >>>> remote: Compressing objects: 32% (37/114) >>>> remote: Compressing objects: 33% (38/114) >>>> remote: Compressing objects: 34% (39/114) >>>> remote: Compressing objects: 35% (40/114) >>>> remote: Compressing objects: 36% (42/114) >>>> remote: Compressing objects: 37% (43/114) >>>> remote: Compressing objects: 38% (44/114) >>>> remote: Compressing objects: 39% (45/114) >>>> remote: Compressing objects: 40% (46/114) >>>> remote: Compressing objects: 41% (47/114) >>>> remote: Compressing objects: 42% (48/114) >>>> remote: Compressing objects: 43% (50/114) >>>> remote: Compressing objects: 44% (51/114) >>>> remote: Compressing objects: 45% (52/114) >>>> remote: Compressing objects: 46% (53/114) >>>> remote: Compressing objects: 47% (54/114) >>>> remote: Compressing objects: 48% (55/114) >>>> remote: Compressing objects: 49% (56/114) >>>> remote: Compressing objects: 50% (57/114) >>>> remote: Compressing objects: 51% (59/114) >>>> remote: Compressing objects: 52% (60/114) >>>> remote: Compressing objects: 53% (61/114) >>>> remote: Compressing objects: 54% (62/114) >>>> remote: Compressing objects: 55% (63/114) >>>> remote: Compressing objects: 56% (64/114) >>>> remote: Compressing objects: 57% (65/114) >>>> remote: Compressing objects: 58% (67/114) >>>> remote: Compressing objects: 59% (68/114) >>>> remote: Compressing objects: 60% (69/114) >>>> remote: Compressing objects: 61% (70/114) >>>> remote: Compressing objects: 62% (71/114) >>>> remote: Compressing objects: 63% (72/114) >>>> remote: Compressing objects: 64% (73/114) >>>> remote: Compressing objects: 65% (75/114) >>>> remote: Compressing objects: 66% (76/114) >>>> remote: Compressing objects: 67% (77/114) >>>> remote: Compressing objects: 68% (78/114) >>>> remote: Compressing objects: 69% (79/114) >>>> remote: Compressing objects: 70% (80/114) >>>> remote: Compressing objects: 71% (81/114) >>>> remote: Compressing objects: 72% (83/114) >>>> remote: Compressing objects: 73% (84/114) >>>> remote: Compressing objects: 74% (85/114) >>>> remote: Compressing objects: 75% (86/114) >>>> remote: Compressing objects: 76% (87/114) >>>> remote: Compressing objects: 77% (88/114) >>>> remote: Compressing objects: 78% (89/114) >>>> remote: Compressing objects: 79% (91/114) >>>> remote: Compressing objects: 80% (92/114) >>>> remote: Compressing objects: 81% (93/114) >>>> remote: Compressing objects: 82% (94/114) >>>> remote: Compressing objects: 83% (95/114) >>>> remote: Compressing objects: 84% (96/114) >>>> remote: Compressing objects: 85% (97/114) >>>> remote: Compressing objects: 86% (99/114) >>>> remote: Compressing objects: 87% (100/114) >>>> remote: Compressing objects: 88% (101/114) >>>> remote: Compressing objects: 89% (102/114) >>>> remote: Compressing objects: 90% (103/114) >>>> remote: Compressing objects: 91% (104/114) >>>> remote: Compressing objects: 92% (105/114) >>>> remote: Compressing objects: 93% (107/114) >>>> remote: Compressing objects: 94% (108/114) >>>> remote: Compressing objects: 95% (109/114) >>>> remote: Compressing objects: 96% (110/114) >>>> remote: Compressing objects: 97% (111/114) >>>> remote: Compressing objects: 98% (112/114) >>>> remote: Compressing objects: 99% (113/114) >>>> remote: Compressing objects: 100% (114/114) >>>> remote: Compressing objects: 100% (114/114), done. >>>> Receiving objects: 0% (1/114) >>>> Receiving objects: 1% (2/114) >>>> Receiving objects: 2% (3/114) >>>> Receiving objects: 3% (4/114) >>>> Receiving objects: 4% (5/114) >>>> Receiving objects: 5% (6/114) >>>> Receiving objects: 6% (7/114) >>>> Receiving objects: 7% (8/114) >>>> Receiving objects: 8% (10/114) >>>> Receiving objects: 9% (11/114) >>>> Receiving objects: 10% (12/114) >>>> Receiving objects: 11% (13/114) >>>> Receiving objects: 12% (14/114) >>>> Receiving objects: 13% (15/114) >>>> Receiving objects: 14% (16/114) >>>> error: object file >>>> .git/objects/21/a28795e9c6fa55f026860e0c2f0b08d1b31611 is empty >>>> fatal: cannot read existing object info >>>> 21a28795e9c6fa55f026860e0c2f0b08d1b31611 >>>> fatal: index-pack failed >>>> >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042) >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1761) >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$400(CliGitAPIImpl.java:72) >>>> at >>>> org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:442) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:575) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource$8.run(AbstractGitSCMSource.java:556) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:367) >>>> at >>>> jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:556) >>>> at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) >>>> at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) >>>> at >>>> jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634) >>>> at >>>> com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277) >>>> at >>>> com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164) >>>> at >>>> jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025) >>>> at hudson.model.ResourceController.execute(ResourceController.java:97) >>>> at hudson.model.Executor.run(Executor.java:429) >>>> Finished: FAILURE >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/jenkinsci-users/b6a89f4c-7b2e-441d-ba26-cf46a762b0af%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> Thanks! >>> Mark Waite >>> >> >> >> -- >> Thanks! >> Mark Waite >> > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b30a6241-1427-46bf-b8a6-34a42b939608%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
