> > I still wonder why there are no options out-of-the-box to: Probably noone needed it badly enough to do it or to pay for it :)
It seems to me that in the Git world, several smaller repos are preferred instead of a big monolithic one. Not sure about you, but in our case the big repo was inherited from a subversion setup. When switching to Git, we kept the centralised workflow and its big repo with lots of obsolete branches, but since then there has been a natural tendency to keep new things in separate repos. While this story may be common in the corporate world, it's probably less so in the more modern open source/startup/smaller agile projects - which I guess are more of a driving force for Jenkins.. Also, while the problem has annoyed me for a long time, it hasn't been a blocker so that I haven't even tried this workaround until recently.. The other reason may be that this would basically involve auto-creating jobs (or something equivalent) on each slave, managed from a central location... at a first glance this differs from the standard ways plugins extending Jenkins. Anyway If I had time I'd have a go at building a plugin - unless anyone started already? On 16 December 2013 20:16, David Gayman <[email protected]> wrote: > Greg, > > Thank you for responding, I am somewhat new to git and had no idea about > the --reference option. Your solution looks like basically the best way to > do this. > > I still wonder why there are no options out-of-the-box to: > > 1) Set up only one cloned repository per project, and > 2) Refuse to check out repositories at the top level (on the machine which > is conducting the build) > > Or, if there are options to do this but I don't know where they are. It > seems like the setup you and I have should be the default! The only thing I > figure is that for generality's sake, each build should have a unique clone > of the repo. > > Thanks. > > *Also, my terminology is too loose - I meant "clone" not "checkout". > > > > > > On Mon, Dec 16, 2013 at 12:59 PM, Gergely Nagy <[email protected]> wrote: > >> Not sure how have the exact same "checkout" multiple times is useful >> (most of my jobs modify the workspace so I need to isolate them from each >> other -> separate workspace == separate "checkouts"). >> >> OTOH, I was also interested to save on disk space when it comes to large >> git repos cloned multiple times. Note that I mean "clone" not "checkout". >> >> So, an easy/safe enough solution to reduce waste of duplicate clones: >> 1) have 1 proper clone of the big repo on each slave - just by create a >> dumb "mirror" job with no build steps, just to keep the clone up to date >> 2) the actual jobs use the "Reference" repo option: although the main >> clone location can be still be the central server, you point to that >> workspace of the mirror job as a reference (see --reference >> https://www.kernel.org/pub/software/scm/git/docs/git-clone.html, and the >> Advanced git plugin option). >> >> My results using this is quite dramatic, the repo (.git) sizes under the >> worker jobs are just a few Mb, even the actually repo is 1.5G+ in my case. >> Also the "clone" for each worker job will just take a fraction of time. >> >> Now I'd like Jenkins (plugin?) to provide a setup like this >> out-of-the-box, eliminating the manual fiddling (I needed to add site-wide >> variables overridden for each slave, etc..). Anyway it's good enough for me >> now. >> >> Of course the actual "checkout" sizes are the same, but I see no >> safe&straightforward way around that, that's just the nature of >> building/testing code. >> So, not sure if this helps but good luck anyway. >> Greg >> >> >> >> On 16 December 2013 17:06, <[email protected]> wrote: >> >>> Has anyone considered a modification to the git plugin (or other SCM >>> plugins), to reduce the total number of checkouts physically required on a >>> slave machine? >>> >>> We run 6 mult-configuration projects (debug and release configurations) >>> on a handful of computers. Each configuration requires one code checkout >>> (2.1 G). That's 2.1*2*6 = 25.2 G on each slave machine, for the source >>> code only. The master build machine requires another 12.6 G, because it >>> checks out additional copies *outside* the scope of build directories (no >>> one knows why). Forget about leaving "Advanced Project Options" > "Restrict >>> where this project can be run" unchecked, because if the build is started >>> on a different computer each time, the code has to be checked out at the >>> top level (potentially adding the 12.6 G to each slave machine, if the load >>> is shared). >>> >>> Perhaps these issues have already been addressed; we run Jenkins version >>> 1.524. >>> >>> Suggestions: >>> a) Live with redundant code checkouts >>> b) Stop using the git plugin, and have my shell script handle code >>> checkout >>> c) Rewrite the git plugin, or add an option, to allow one checkout per >>> project on a slave. Technically this is the minimum number of checkouts >>> allowed, because each project is allowed to access a unique sha1 hash. >>> d) Add an option to the Jenkins core to disregard the top-level code >>> checkouts (which I gather are done for some kind of consistency check...we >>> can live without that check I think). >>> e) I don't know Java and don't have much free time, otherwise I would >>> write a completely new git plugin. >>> >>> Thanks for any suggestions, especially if I am missing something obvious >>> here. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Jenkins Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Jenkins Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jenkinsci-users/IwgQSKqrC6g/unsubscribe >> . >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
