[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite commented on JENKINS-51542 Re: Git checkout is slower than the command line execution As far as I can tell, must users gain significantly more from using a reference repository and an intentionally narrow refspec than from an isolated shallow clone. Those are just my observations, not anything that I've rigorously compared in a controlled environment. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.5791.1582591320323%40Atlassian.JIRA.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title
Marcelo Oliveira edited a comment on JENKINS-51542
Re: Git checkout is slower than the command line execution
FYI, I was also having this problem and it boiled down to shallow depth = 1 clone, not sure if it's the same but it may help others.3m30s with depth =1{noformat}00:00:02.788 > /usr/bin/git fetch --tags --progress --depth=1 -- g...@git.xyz.com:repo.git +refs/heads/*:refs/remotes/origin/* # timeout=150 00:03:26.147 > /usr/bin/git config remote.origin.url g...@git.xyz.com:repo.git # timeout=60{noformat}40s normal fetch {noformat}00:00:02.472 > /usr/bin/git fetch --tags --progress -- g...@git.xyz.com:apps/repo.git +refs/heads/*:refs/remotes/origin/* # timeout=150 00:00:42.424 > /usr/bin/git config remote.origin.url g...@git.xyz.com:apps/repo.git # timeout=60 {noformat}*Node:*git version 2.23.0git-lfs/2.10.0 (GitHub; darwin amd64; go 1.13.6)*Jenkins:*git-client 3.1.1git-plugin: 4.1.1.*Git server:*GitLab Enterprise Edition 12.7.2-ee Maybe the server spends more time figuring out the shallow copy than just send the pack files?
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.5767.1582589280928%40Atlassian.JIRA.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title
Marcelo Oliveira edited a comment on JENKINS-51542
Re: Git checkout is slower than the command line execution
FYI, I was also having this problem and it boiled down to shallow depth = 1 clone, not sure if it's the same but it may help others. 3m30s ~3m24s with depth =1{noformat}00:00:02.788 > /usr/bin/git fetch --tags --progress --depth=1 -- g...@git.xyz.com:repo.git +refs/heads/*:refs/remotes/origin/* # timeout=150 00:03:26.147 > /usr/bin/git config remote.origin.url g...@git.xyz.com:repo.git # timeout=60{noformat} ~ 40s normal fetch{noformat}00:00:02.472 > /usr/bin/git fetch --tags --progress -- g...@git.xyz.com:apps/repo.git +refs/heads/*:refs/remotes/origin/* # timeout=15000:00:42.424 > /usr/bin/git config remote.origin.url g...@git.xyz.com:apps/repo.git # timeout=60 {noformat}*Node:*git version 2.23.0git-lfs/2.10.0 (GitHub; darwin amd64; go 1.13.6)*Jenkins:*git-client 3.1.1git-plugin: 4.1.1.*Git server:*GitLab Enterprise Edition 12.7.2-eeMaybe the server spends more time figuring out the shallow copy than just send the pack files?
Add Comment
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.5769.1582589280974%40Atlassian.JIRA.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Marcelo Oliveira commented on JENKINS-51542 Re: Git checkout is slower than the command line execution FYI, I was also having this problem and it boiled down to shallow depth = 1 clone, not sure if it's the same but it may help others. 3m30s with depth =1 00:00:02.788 > /usr/bin/git fetch --tags --progress --depth=1 -- g...@git.xyz.com:repo.git +refs/heads/*:refs/remotes/origin/* # timeout=150 00:03:26.147 > /usr/bin/git config remote.origin.url g...@git.xyz.com:repo.git # timeout=60 40s normal fetch 00:00:02.472 > /usr/bin/git fetch --tags --progress -- g...@git.xyz.com:apps/repo.git +refs/heads/*:refs/remotes/origin/* # timeout=150 00:00:42.424 > /usr/bin/git config remote.origin.url g...@git.xyz.com:apps/repo.git # timeout=60 Node: git version 2.23.0 git-lfs/2.10.0 (GitHub; darwin amd64; go 1.13.6) Jenkins: git-client 3.1.1 git-plugin: 4.1.1. Git server: GitLab Enterprise Edition 12.7.2-ee Maybe the server spends more time figuring out the shallow copy than just send the pack files? Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Vitaly Gorbunov edited a comment on JENKINS-51542 Re: Git checkout is slower than the command line execution Actually I just realized that on other instance with git version 1.8.3.1 it works fast.The difference is that it prints tags to stdout much faster You can disregard my comments - when I upgrade to git 2.16.5 it become fast again. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.3675.1562491560416%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Vitaly Gorbunov edited a comment on JENKINS-51542 Re: Git checkout is slower than the command line execution Actually I just realized that on other instance with git version 1.8.3.1 it works fast. The difference is that it prints tags to stdout much fasterYou can disregard my comments - when I upgrade upgraded to git 2.16.5 it become became fast again. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.3677.1562491560466%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Vitaly Gorbunov commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Actually I just realized that on other instance with git version 1.8.3.1 it works fast. The difference is that it prints tags to stdout much faster Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.3663.1562490300279%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title
Vitaly Gorbunov edited a comment on JENKINS-51542
Re: Git checkout is slower than the command line execution
For me the issue is with amount of tags in our repository (7k)I've created repository with 10k tags to reproduce the issue - [https://github.com/vitgorbunov/lots-of-tags] git version 2.14. 1 2 git client plugin version 3.0.0-rc git plugin version 4.0.1-rc-rc3051.45f40fc87c7e I'm considering to just use shell commands, but I will miss git commits summary on build page in this case. {code:java}#clone through git plugin00:00:00.044 Cloning repository https://github.com/vitgorbunov/lots-of-tags.git00:00:00.044 > git init /tmp/jenkins-b280d87a/workspace/vgr_slow_git_test # timeout=1000:00:00.049 Fetching upstream changes from https://github.com/vitgorbunov/lots-of-tags.git00:00:00.050 > git --version # timeout=1000:00:00.053 > git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/* # timeout=1000:06:50.116 > git config remote.origin.url https://github.com/vitgorbunov/lots-of-tags.git # timeout=1000:06:50.116 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=1000:06:50.119 > git config remote.origin.url https://github.com/vitgorbunov/lots-of-tags.git # timeout=1000:06:50.127 Fetching upstream changes from https://github.com/vitgorbunov/lots-of-tags.git00:06:50.127 > git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/* # timeout=1000:06:52.919 > git rev-parse refs/remotes/origin/master^{commit} # timeout=1000:06:52.924 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=1000:06:52.931 Checking out Revision afaccdc4e7350a09bca383c2b9ac458578e1a34b (refs/remotes/origin/master)00:06:52.933 > git config core.sparsecheckout # timeout=1000:06:52.936 > git checkout -f afaccdc4e7350a09bca383c2b9ac458578e1a34b # timeout=1000:06:52.950 Commit message: "1"00:06:52.953 > git rev-list --no-walk afaccdc4e7350a09bca383c2b9ac458578e1a34b # timeout=10#Using shell command00:06:53.008 [vgr_slow_git_test] $ /bin/sh -xe /tmp/jenkins3386321295493424405.sh00:06:53.011 + git clone https://github.com/vitgorbunov/lots-of-tags.git lots-of-tags-cloned00:06:53.013 Cloning into 'lots-of-tags-cloned'...00:06:54.464 00:06:54.464 real 0m1.453s00:06:54.464 user 0m0.398s00:06:54.464 sys 0m0.281s{code} Extract from thread dump{code:java}"git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/*: stderr copier" #27955 prio=5 os_prio=0 tid=0x7f57b800b800 nid=0x12fa runnable [0x7f57f2c18000] java.lang.Thread.State: RUNNABLEat java.io.FileInputStream.readBytes(Native Method)at java.io.FileInputStream.read(FileInputStream.java:255)at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)at java.io.BufferedInputStream.read(BufferedInputStream.java:345)- locked <0x00076fb22798> (a java.lang.UNIXProcess$ProcessPipeInputStream)at java.io.FilterInputStream.read(FilterInputStream.java:107)at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)"git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/*: stdout copier" #27953 prio=5 os_prio=0 tid=0x7f57b8001800 nid=0x12f8 runnable [0x7f57f8e9d000] java.lang.Thread.State: RUNNABLEat java.io.FileInputStream.readBytes(Native Method)at java.io.FileInputStream.read(FileInputStream.java:255)at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)at
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title
Vitaly Gorbunov edited a comment on JENKINS-51542
Re: Git checkout is slower than the command line execution
For me the issue is with amount of tags in our repository (7k)I've created repository with 10k tags to reproduce the issue - [https://github.com/vitgorbunov/lots-of-tags]git version 2.14.1git client plugin version 3.0.0-rcgit plugin version 4.0.1-rc-rc3051.45f40fc87c7e I'm consider considering to just use shell commands, but I will miss git commits summary on build page in this case. {code:java}#clone through git plugin00:00:00.044 Cloning repository https://github.com/vitgorbunov/lots-of-tags.git00:00:00.044 > git init /tmp/jenkins-b280d87a/workspace/vgr_slow_git_test # timeout=1000:00:00.049 Fetching upstream changes from https://github.com/vitgorbunov/lots-of-tags.git00:00:00.050 > git --version # timeout=1000:00:00.053 > git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/* # timeout=1000:06:50.116 > git config remote.origin.url https://github.com/vitgorbunov/lots-of-tags.git # timeout=1000:06:50.116 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=1000:06:50.119 > git config remote.origin.url https://github.com/vitgorbunov/lots-of-tags.git # timeout=1000:06:50.127 Fetching upstream changes from https://github.com/vitgorbunov/lots-of-tags.git00:06:50.127 > git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/* # timeout=1000:06:52.919 > git rev-parse refs/remotes/origin/master^{commit} # timeout=1000:06:52.924 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=1000:06:52.931 Checking out Revision afaccdc4e7350a09bca383c2b9ac458578e1a34b (refs/remotes/origin/master)00:06:52.933 > git config core.sparsecheckout # timeout=1000:06:52.936 > git checkout -f afaccdc4e7350a09bca383c2b9ac458578e1a34b # timeout=1000:06:52.950 Commit message: "1"00:06:52.953 > git rev-list --no-walk afaccdc4e7350a09bca383c2b9ac458578e1a34b # timeout=10#Using shell command00:06:53.008 [vgr_slow_git_test] $ /bin/sh -xe /tmp/jenkins3386321295493424405.sh00:06:53.011 + git clone https://github.com/vitgorbunov/lots-of-tags.git lots-of-tags-cloned00:06:53.013 Cloning into 'lots-of-tags-cloned'...00:06:54.464 00:06:54.464 real 0m1.453s00:06:54.464 user 0m0.398s00:06:54.464 sys 0m0.281s{code} Extract from thread dump{code:java}"git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/*: stderr copier" #27955 prio=5 os_prio=0 tid=0x7f57b800b800 nid=0x12fa runnable [0x7f57f2c18000] java.lang.Thread.State: RUNNABLEat java.io.FileInputStream.readBytes(Native Method)at java.io.FileInputStream.read(FileInputStream.java:255)at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)at java.io.BufferedInputStream.read(BufferedInputStream.java:345)- locked <0x00076fb22798> (a java.lang.UNIXProcess$ProcessPipeInputStream)at java.io.FilterInputStream.read(FilterInputStream.java:107)at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)"git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/*: stdout copier" #27953 prio=5 os_prio=0 tid=0x7f57b8001800 nid=0x12f8 runnable [0x7f57f8e9d000] java.lang.Thread.State: RUNNABLEat java.io.FileInputStream.readBytes(Native Method)at java.io.FileInputStream.read(FileInputStream.java:255)at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)at
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title
Vitaly Gorbunov commented on JENKINS-51542
Re: Git checkout is slower than the command line execution
For me the issue is with amount of tags in our repository (7k) I've created repository with 10k tags to reproduce the issue - https://github.com/vitgorbunov/lots-of-tags git version 2.14.1 git client plugin version 3.0.0-rc git plugin version 4.0.1-rc-rc3051.45f40fc87c7e I'm consider to just use shell commands, but I will miss git commits summary on build page in this case.
#clone through git plugin
00:00:00.044 Cloning repository https://github.com/vitgorbunov/lots-of-tags.git
00:00:00.044 > git init /tmp/jenkins-b280d87a/workspace/vgr_slow_git_test # timeout=10
00:00:00.049 Fetching upstream changes from https://github.com/vitgorbunov/lots-of-tags.git
00:00:00.050 > git --version # timeout=10
00:00:00.053 > git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/* # timeout=10
00:06:50.116 > git config remote.origin.url https://github.com/vitgorbunov/lots-of-tags.git # timeout=10
00:06:50.116 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
00:06:50.119 > git config remote.origin.url https://github.com/vitgorbunov/lots-of-tags.git # timeout=10
00:06:50.127 Fetching upstream changes from https://github.com/vitgorbunov/lots-of-tags.git
00:06:50.127 > git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/* # timeout=10
00:06:52.919 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
00:06:52.924 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
00:06:52.931 Checking out Revision afaccdc4e7350a09bca383c2b9ac458578e1a34b (refs/remotes/origin/master)
00:06:52.933 > git config core.sparsecheckout # timeout=10
00:06:52.936 > git checkout -f afaccdc4e7350a09bca383c2b9ac458578e1a34b # timeout=10
00:06:52.950 Commit message: "1"
00:06:52.953 > git rev-list --no-walk afaccdc4e7350a09bca383c2b9ac458578e1a34b # timeout=10
#Using shell command
00:06:53.008 [vgr_slow_git_test] $ /bin/sh -xe /tmp/jenkins3386321295493424405.sh
00:06:53.011 + git clone https://github.com/vitgorbunov/lots-of-tags.git lots-of-tags-cloned
00:06:53.013 Cloning into 'lots-of-tags-cloned'...
00:06:54.464
00:06:54.464 real 0m1.453s
00:06:54.464 user 0m0.398s
00:06:54.464 sys 0m0.281s
Extract from thread dump
"git fetch --tags --progress https://github.com/vitgorbunov/lots-of-tags.git +refs/heads/*:refs/remotes/origin/*: stderr copier" #27955 prio=5 os_prio=0 tid=0x7f57b800b800 nid=0x12fa runnable [0x7f57f2c18000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00076fb22798> (a java.lang.UNIXProcess$ProcessPipeInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60)
"git
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Antony Gelberg commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Mark Waite Thanks for the response. I should have been more detailed in my comment. I don't know about the difference between fetch and clone, but I do know that on a fresh workspace, it can take 10-15 minutes when the equivalent shell command takes under half a minute (I didn't bother measuring, the order of magnitude is so great). On subsequent builds, it's acceptable aka normal. Unfortunately, our repo and configuration are (mostly) private, but I'm happy to do a screenshare with you to demonstrate, perhaps we can do some debugging online. This is the second project I've worked on that has experienced this issue, so I don't see it as being hard to reproduce. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.14440.1562074140211%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite commented on JENKINS-51542 Re: Git checkout is slower than the command line execution bhargav home and Antony Gelberg while I appreciate "still slow" as an answer, that isn't enough to persuade me to investigate further. The last time that users expressed their strongly held opinion that git clone is significantly faster than git fetch with the same repository and the same environment, I spent several hours running a series of benchmark comparisons. The benchmark comparisons showed no significant difference between git clone and git fetch. The git client plugin uses git fetch instead of git clone because there are optimizations which can be performed with git fetch which are much more difficult to perform with git clone, especially across the wide range of command line git versions supported by the plugin. I'd be much more persuaded that there is a significant difference between git fetch and git clone if comparative data were presented which showed the difference. That comparative data needs to be documented well enough that others can duplicate the configuration in order to see the problem. Most often, reports of "slow clone" are best addressed by techniques described in the "Git in the Large" presentation from Jenkins World 2017. The technique presented there include: Use reference repositories to reduce clone time and clone disc space use Use narrow refspecs to reduce clone time and clone disc space Use shallow clone to reduce clone time and clone disc space Use sparse checkout to reduce working directory disc space Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title bhargav home commented on JENKINS-51542 Re: Git checkout is slower than the command line execution I am using git version 2.20.0 and still, it's slow while checking out. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.190925.1527240014000.14181.1562055900263%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Antony Gelberg commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Still really slow using git 2.21.0 on Windows Server 2012 R2 slave. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Thanks for the update Jonathan Rogers! Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Jonathan Rogers commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Mark Waite: You are certainly correct that "git fetch" has changed behavior since 1.7.1. After I last commented, I noticed that that version is eight years old so I decided to try a newer one. For me, the difference in checkout time between "git clone" and the sequence of commands issued by the git plugin mostly or completely disappeared with git 2.5. Thanks. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Jonathan Rogers there are many issues in command line git 1.7.1 that are resolved in later versions of git. Shallow clone (for example) is not fully supported in git 1.7.1. It may be that reference repositories are also not supported in git 1.7.1 using git fetch. You might consider enabling the JGit implementation for that repository and use JGit instead of command line git. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Fabian Holler commented on JENKINS-51542 Re: Git checkout is slower than the command line execution We also run into the issue with pipeline scripts + git repositories + a local reference repository. During initial checkout for the pipeline script, the "lightweight checkout" options increases the checkout a lot. With lightweight checkout disabled + reference repo it takes ~5sec. With lightweight checkout enabled + reference repo it takes ~15min. The checkout done via checkout(scm), takes often ~15min despite the reference repository. If I do a "git clone --reference" on the same host, same filesystem it takes ~5sec. Freestyle Git checkouts with a reference repository a fast as expected. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Jonathan Rogers commented on JENKINS-51542 Re: Git checkout is slower than the command line execution I have done some more experimenting with git commands. For my use case, the difference between using the git plugin to do a checkout vs. using "git clone" is primarily about the reference repository. AFAICT, the commands issued by the git plugin are not able to take advantage of a reference repository. Unfortunately, how reference repositories are configured isn't very well documented. The git-clone man page mentions ".git/objects/info/alternates" and after issuing a "git clone" command with the "–reference" option, that file contains the path of the reference repository's "objects" directory. After issuing such a command, I get a new repository in which the ".git" directory only occupies a couple of megabytes. Also AFAICT, the git plugin directly writes the correct path to ".git/objects/info/alternates" before issuing the "git fetch" command. However, "git fetch" does not seem to take advantage of the reference repository, since the ".git" directory in the new repository is several hundred megabytes. The git plugin's reference repository option simply has no effect. Since the the "git fetch" man page says nothing about reference repositories or alternates, I can't tell if that command is behaving correctly. BTW, I'm using git 1.7.1. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Jonathan Rogers it only uses git merge when requested to perform a pre-build merge. It uses a detached HEAD checkout unless specifically configured to checkout with a branch name. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Jonathan Rogers commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Does the Jenkins git plugin call "git merge"? I've only seen "git ls-remote", "git rev-parse", "git config", "git fetch" and "git checkout". Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite commented on JENKINS-51542 Re: Git checkout is slower than the command line execution Jonathan Rogers the textual description of git clone is that it is the combination of git fetch and git merge. However, we've also had reports from a few Jenkins users that they also see faster performance if they use git clone instead of the git fetch which is used by the git plugin. When I attempted to duplicate the performance difference in that case, I was unable to do so. At that time, I consistently found comparable performance when comparing git clone and git fetch + git merge. That likely means that I don't understand the specific details which cause clone to be faster than fetch. Unfortunately, the connection between fetch and the use cases of the git plugin are strong enough that I don't think there will ever be a way to call git clone directly from the plugin. The change from clone to fetch would break many different use cases. As one example, I've seen much better performance if I specify a narrow refspec to the git plugin. The multibranch pipeline code uses that technique to limit a clone to a single branch, rather than cloning all branches in the repository. The narrow refspec limits the amount of information sent from the remote to the local to only the specific named branch. Unfortunately, the git clone command doesn't accept a refspec as an argument. There are command line arguments that will allow adding more refspecs, but there is no way to limit the refspec from the clone command line. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Jonathan Rogers commented on JENKINS-51542 Re: Git checkout is slower than the command line execution I am also able to get much better performance from a custom "git clone" command than what the Jenkins git plugin does. Specifically, I use "git clone --depth 1 --reference /reference/path git://git.example.com/thing.git", which takes 20 to 30s for a repository of about 900MiB. The Jenkins configuration which attempts to duplicate this behavior usually takes over 2.5 minutes to check out the code. The strange thing is that sometimes it only takes 30s. I haven't been able to determine why the time varies so much, but it seems like it may be quicker immediately after the reference repository was updated. It seems the Jenkins git plugin never emits a "git clone" command, but rather "git fetch" followed by "git checkout". Perhaps there's no way to exactly duplicate "git clone" with such a sequence? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite commented on JENKINS-51542 Re: Git checkout is slower than the command line execution And yet the second call to `git fetch` completes in 3 seconds (09:54:15 -> 09:54:18) while the first call to `git fetch` requires 60 seconds. The second call to `git fetch` does not seem to be a dramatic contributor to slower performance. Can you provide more details that compare the same commands from a command line? Are the same commands also using the same file system? Are there other differences which affect performance? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Mark Waite assigned an issue to Unassigned Jenkins / JENKINS-51542 Git checkout is slower than the command line execution Change By: Mark Waite Assignee: Mark Waite Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-51542) Git checkout is slower than the command line execution
Title: Message Title Oliver Pereira created an issue Jenkins / JENKINS-51542 Git checkout is slower than the command line execution Issue Type: Bug Assignee: Mark Waite Components: git-plugin Created: 2018-05-25 09:20 Environment: Development Priority: Major Reporter: Oliver Pereira The git checkout is much slower even after using reference repositories, shallow clones etc. Running the same commands via the command line is much faster. I am using the latest version of all the plugins. If you look at the below output, it runs the fetch more than once. 09:53:13 Cloning the remote Git repository 09:53:13 Using shallow clone 09:53:13 Avoid fetching tags 09:53:13 Cloning repository git@xxx:/xxx.git 09:53:13 > git init /srv/jenkins/workspace/shared-buck-2-master # timeout=10 09:53:14 Using reference repository: /var/lib/jenkins/reference-repositories/xx.git 09:53:14 Fetching upstream changes from git@x:/xxx.git 09:53:14 > git --version # timeout=10 09:53:14 > git fetch --no-tags --progress git@xx:/xxx.git +refs/heads/*:refs/remotes/xxx/* --depth=1 09:54:15 > git config remote.xxx.url git@:x/.git # timeout=10 09:54:15 > git config --add remote..fetch +refs/heads/*:refs/remotes/xxx/* # timeout=10 09:54:15 > git config remote.xxx.url git@:x/xxx.git # timeout=10 09:54:15 Fetching upstream changes from git@:/xxx.git 09:54:15 > git fetch --no-tags --progress git@:x/.git +refs/heads/*:refs/remotes/xx/* --depth=1 09:54:18 > git rev-parse
