On my previous project we indeed had some binaries in the repos, which made cloning a pain. However, we now have very new, very small repos, which contain nothing but the code and some configuration and documentation. So no binaries. I will look into the file system though, because Jenkins is running inside a VM.
About the git settings: too bad, I think I will try and convince the others to consolidate into one repository, because there are many issues related to the multi-repo setup we chose... On Friday, February 6, 2015 at 2:34:42 PM UTC+1, Mark Waite wrote: > > > > On Fri, Feb 6, 2015 at 5:50 AM, Titus Nachbauer <[email protected] > <javascript:>> wrote: > >> Thank you for the kind answer. I saw many of the git plugin options >> mentioned here and there, but since I am using the Multiple SCM plugin, >> there is no place to select these options. Would you know where to find >> them and if it is possible with this plugin? >> >> > I don't know how to access all the capabilities of the git plugin from the > multiple SCM plugin. > > >> About the cloning: I know that in general the git plugin actually does >> not use clone, but init and then fetch. In my particular case you are >> right: it only fetches and checks out the latest changes. However, this is >> relatively slow. The whole process takes 60 seconds (30 s for the master, >> 30 s for the workspace), where the rest of the build takes 2. This is not a >> huge issue now, I just want to make sure it does not become one later. >> >> > Wow, that is impressive. An incremental update takes 30 seconds per > workspace? That seems like a very long time for an incremental update. > > That may indicate that people are pushing large binaries into your git > repository. That has been a "bad thing" for me at two different > employers. When large binaries are pushed into git repositories, they > become bulky and more difficult to manage. Git can support large > repositories (thankfully, since I have one that is over 9 GB now), but it > works best when it manages source code rather than binaries. The Linux > kernel (millions of files, multiple commits per hour, 24 hours a day, 7 > days a week) is only a few hundred megabytes, and clones (without checkout) > from github to my local Ubuntu machine in about a minute when I use a > reference repository. > > It might instead indicate that you're on a Windows machine, or on a > virtual machine with a slower file system. > > Good luck! > Mark Waite > > >> On Friday, February 6, 2015 at 1:19:17 PM UTC+1, Mark Waite wrote: >>> >>> >>> >>> On Fri, Feb 6, 2015 at 3:32 AM, Titus Nachbauer <[email protected]> >>> wrote: >>> >>>> We have a Jenkins job which uses Multiple SCM to clone 5 repositories >>>> and then builds it using a gradle script. There are two things that are >>>> slowing us down: >>>> >>>> 1. A build is triggered on every repository on every change. That >>>> is fine, but it currently means that every time one of the repositories >>>> changes, all repositories are cloned again. Is there a way to make sure >>>> only the changed repo is cloned? >>>> >>>> I don't understand that statement. When my multi-configuration jobs >>> run (single repository), they only clone a new copy of the repository the >>> first time the build is executed on a particular slave or when the slave >>> workspace is "wiped". After the first build, subsequent builds fetch only >>> what has changed since the last build. >>> >>> Can you clarify further? >>> >>> >>>> >>>> 1. Since it is a multi-configuration job, the clone is executed >>>> twice, once in the parent workspace and once in the configuration >>>> workspace. Now as I understand it this is the expected behaviour, but >>>> is >>>> there a way to change this and only clone in one of those or just copy >>>> the >>>> cloned workspace? >>>> >>>> There is no way that I know to run a multi-configuration job without >>> cloning once in the configuration workspace. >>> >>>> Also would there be a way to tell Jenkins to normally pull the repos >>>> with a hart reset and only clone on changes of .gitignore? >>>> >>>> The git plugin allows you to ignore commits based on file name patterns >>> or on user name patterns. You might investigate that option as a way to >>> reduce the number of times a job is started. >>> >>> If you have a large repository, you may also be able to improve cloning >>> speed significantly by using a "reference" repository. A reference >>> repository is a bare repository clone of the original repository. When git >>> is told to use a reference repository, it reuses the content available in >>> the reference repository rather than downloading it. >>> >>> You may also be able to improve cloning speed by using a "shallow >>> clone". Both options are available from the git plugin advanced options. >>> >>> Refer to http://blog.cloudbees.com/2014/09/advanced-git-with- >>> jenkins.html for a blog posting that describes the topic further. >>> >>> >>>> This is a duplicate of my question on Stackoverflow >>>> <http://stackoverflow.com/questions/28346410/speed-up-jenkins-cloning-git-repositories-with-multiple-scm-plugin-and-multi-con>, >>>> >>>> it seems there is not much activity on the Jenkins front there. What is >>>> the >>>> best place to ask professional Jenkins questions? >>>> >>> I find this mailing list is the best place to ask Jenkins user >>> questions. There are also several books available which describe Jenkins. >>> There are companies (like Cloudbees) which provide enterprise licensed and >>> supported versions of Jenkins as well. >>> >>> Mark Waite >>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Jenkins Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/jenkinsci-users/80d1217d-0b7c-42d2-88c8- >>>> 40f7138f27b5%40googlegroups.com >>>> <https://groups.google.com/d/msgid/jenkinsci-users/80d1217d-0b7c-42d2-88c8-40f7138f27b5%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Thanks! >>> Mark Waite >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jenkinsci-users/e8ffb5ee-8e6e-4608-9be3-92065ad5c128%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jenkinsci-users/e8ffb5ee-8e6e-4608-9be3-92065ad5c128%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Thanks! > Mark Waite > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b066d050-634e-4cea-bdc0-b40810f6155d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
