Ok circling back to this. Thanks for the suggestion about using a local repo Marques. TLDR: it worked!
Long form: I created a bogus local repo (as a GKE PVC). I mounted the PVC on both the server and the agent as ReadOnlyMany (only way to get a PVC mounted on multiple pods). With this, I configured the pipeline git material to point to the bogus local repo. This allows me to bypass the pipeline material restriction which requires a pipeline to have some sort of material. Since the same disk is also mounted on the agent, I was able to clone the bogus repo. Now, i've successfully bypassed the material restriction of the pipeline and the CI builds on the agents does not fail! Now for the fun part, how do I get my actual large repo in the agent? So I dug through the jenkins source code to see how they handle reference repos. This is what they do (which is what i'm doing as a job within the pipeline) $ git init <repo> $ cd <repo> $ echo "/path/to/reference/repo/.git/objects" > .git/objects/info/alternates $ git config remote.origin.url <repo url e.g https://github.com/gocd/somerepo> $ git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master $ git fetch --no-tags --progress <repo url e.g https://github.com/gocd/somerepo> +refs/heads/master:refs/remotes/origin/master $ git checkout master # Now I have a checked out repo & can run my builds! You may ask instead of using a bogus repo, why not just mount the actual reference repo on the server and agent? Well, the reference repo needs to be updated (daily). Since the reference repo is mounted as ReadOnlyMany, it can't be updated when it is in use. Update entails: - Creating a new PV/PVC - Updating pod template to use the new PV/PVC - Delete the old PV/PVC The agents are elastic - a build creates a new pod - so every new pod get the latest pod template with the latest PVC. The issue is with updating the PVC for the server. The server is a long running pod and the PV/PVC can not be swapped while it is running. It needs to be recreated which is quite disruptive to service. So I decided to mount a PV which does not change frequently hence the bogus repo. IMHO, the condition that a pipeline needs to have a material is too restrictive. There are limited available materials and those materials don't cover all CI use cases. I think either have material that supports a robust amount of CI use cases OR make materials optional to pipelines. Thank again for the suggestion Marques! On Monday, 10 May 2021 at 20:49:41 UTC-7 [email protected] wrote: > Is creating a new PV and provisioning that a feasible option? > > That's what I do on my Illumos build zones for some use cases - I use a > ZFS snapshot, mount it into a Zone and run specific build experiments. It > saves me a lot of time since I can avoid cloning gigabytes of code even > from the same baremetal server. > > On Tue, May 11, 2021 at 10:36 AM Marques Lee <[email protected]> > wrote: > >> Thank you Hussein for that, glad to help. >> >> You may have seen this article here: >> >> https://medium.com/@Sushil_Kumar/readwritemany-persistent-volumes-in-google-kubernetes-engine-a0b93e203180 >> >> I think this person works around limitations on readwritemany by creating >> separate PVs fir read and write backed by the same NFS disk. Not sure if >> that’s helpful or not or worth the extra infrastructure. >> >> On Mon, May 10, 2021 at 7:28 PM HUSSEIN KADIRI <[email protected]> wrote: >> >>> Thank you Marques, I really appreciate your help and support. You've >>> been very responsive. >>> I'm looking for an alternative to Jenkins. Unfortunately we have really >>> large repos so most existing CI tools don't work out of the box because of >>> the clone issues. >>> I really like GoCD and have enjoyed creating a POC for it. I don't want >>> to drop it just yet. I'll continue to look for a workaround. >>> >>> Thanks again >>> >>> >>> >>> On Mon, May 10, 2021 at 7:12 PM Marques Lee <[email protected]> >>> wrote: >>> >>>> Ok, fair enough :). Let us know if we can help in any way. >>>> >>>> On Mon, May 10, 2021 at 7:06 PM HUSSEIN KADIRI <[email protected]> >>>> wrote: >>>> >>>>> Let me explore the local clone option some more >>>>> $ git clone /local/path/to/bare/repo >>>>> >>>>> Thanks for the tip >>>>> On Monday, 10 May 2021 at 19:01:34 UTC-7 HUSSEIN KADIRI wrote: >>>>> >>>>>> You're right the agent won't need read write. The volume is what >>>>>> needs to be setup as read write. See doc >>>>>> <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes> >>>>>> >>>>>> Write - would be the "cron" updating the repo >>>>>> Read - would be the agents "cloning" >>>>>> Reading and Writing can't happen simultaneously. >>>>>> >>>>>> On Monday, 10 May 2021 at 18:55:21 UTC-7 Marques Lee wrote: >>>>>> >>>>>>> Typo — “why would the agents need read/write” was what I meant >>>>>>> >>>>>>> On Mon, May 10, 2021 at 6:53 PM Marques Lee < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Pardon my lack of familiarity with GKE, my question may be >>>>>>>> extremely naive - if there’s only process that needs to write to the >>>>>>>> mount >>>>>>>> (the cron that performs the fetch to keep the code updated), then by >>>>>>>> would >>>>>>>> the agents need read/write on the volume? Isn’t read-only enough to >>>>>>>> clone >>>>>>>> for a pipeline run? >>>>>>>> >>>>>>>> On Mon, May 10, 2021 at 6:48 PM HUSSEIN KADIRI <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hmm, we setup GoCD on kubernetes so the server is a kubernetes >>>>>>>>> deployment. >>>>>>>>> >>>>>>>>> We are on GKE . Yes we can mount a volume but GKE PVC are only >>>>>>>>> ReadWriteOnce (One pod mounted to read and write) or ReadOnlyMany >>>>>>>>> (Multiple >>>>>>>>> pods mounted and reading only from the volume). >>>>>>>>> >>>>>>>>> Your recommendation requires a ReadWriteMany setup which is not >>>>>>>>> possible in GKE. >>>>>>>>> >>>>>>>>> On Mon, May 10, 2021, 6:43 PM Marques Lee < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hmm, unfortunately I don’t believe the git material in GoCD has >>>>>>>>>> native support for reference repos. >>>>>>>>>> >>>>>>>>>> Could the server not run a cron to do git fetch —all on the bare >>>>>>>>>> repo to keep it updated? >>>>>>>>>> >>>>>>>>>> Then all agents would pick up new changes so long as the volume >>>>>>>>>> holding the bare repo is mounted to agents. >>>>>>>>>> >>>>>>>>>> Otherwise, yeah, GoCD may not be a good fit for you until we >>>>>>>>>> build in support for —reference-repo if that’s a deal breaker. >>>>>>>>>> >>>>>>>>>> On Mon, May 10, 2021 at 6:37 PM HUSSEIN KADIRI <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I have elastic agents so the cron route would not be feasible. >>>>>>>>>>> >>>>>>>>>>> I mount a reference repo as a K8s PVC. >>>>>>>>>>> >>>>>>>>>>> I want to do a git clone <url> --reference-repo <path to my >>>>>>>>>>> mounted reference repo>. >>>>>>>>>>> >>>>>>>>>>> Can the git material support or be modified to accept reference >>>>>>>>>>> repo paths? >>>>>>>>>>> >>>>>>>>>>> This is kind of a deal breaker if it can't >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, May 10, 2021, 6:31 PM Marques Lee < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hmm. Well I suppose you could git clone —bare —mirror to a >>>>>>>>>>>> known location and then: >>>>>>>>>>>> >>>>>>>>>>>> 1) create git materials with file system paths for the url >>>>>>>>>>>> instead of http/ssh >>>>>>>>>>>> 2) run a cron to keep the ref repo updated >>>>>>>>>>>> >>>>>>>>>>>> Would that work? >>>>>>>>>>>> On Mon, May 10, 2021 at 6:25 PM HUSSEIN KADIRI < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> My repo is too big, it can't be cloned by itself. it needs a >>>>>>>>>>>>> reference repo. >>>>>>>>>>>>> >>>>>>>>>>>>> Is there a way to configure a reference repo ? >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, May 10, 2021, 6:23 PM Marques Lee < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Git materials support shallow clone. I think you need to >>>>>>>>>>>>>> expand the advance tab to see the option. It’s of course in >>>>>>>>>>>>>> cruise-config.xml as well as the various pipelines-as-config >>>>>>>>>>>>>> syntaxes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The workspace isn’t recloned every time either. If it exists >>>>>>>>>>>>>> on disk, it gets updated via fetch. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, May 10, 2021 at 4:29 PM HUSSEIN KADIRI < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> Sometimes a git repo is large and so it's not efficient/fast >>>>>>>>>>>>>>> to do a git clone. One would have to use a reference repo. For >>>>>>>>>>>>>>> all the >>>>>>>>>>>>>>> parts that require git (config repository, git material, etc), >>>>>>>>>>>>>>> is it >>>>>>>>>>>>>>> possible to have more complex git workflows - utilizing a >>>>>>>>>>>>>>> reference repo, >>>>>>>>>>>>>>> shallow clone - since a simple git clone is not always >>>>>>>>>>>>>>> possible? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>> Google Groups "go-cd" group. >>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>> from it, send an email to [email protected]. >>>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAFD%2B7Dm2Ah3rAMzkt8j9pM_LoAKOn%3DgK0KMqNW-7dm8FuQOpyA%40mail.gmail.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAFD%2B7Dm2Ah3rAMzkt8j9pM_LoAKOn%3DgK0KMqNW-7dm8FuQOpyA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>> You received this message because you are subscribed to a >>>>>>>>>>>>>> topic in the Google Groups "go-cd" group. >>>>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>>>> https://groups.google.com/d/topic/go-cd/wEK4-BYReb8/unsubscribe >>>>>>>>>>>>>> . >>>>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAPKX9jatvS8kBrhek9meEdOJpus6Hb%2BpLBnWjcxNoQK97FXn7A%40mail.gmail.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAPKX9jatvS8kBrhek9meEdOJpus6Hb%2BpLBnWjcxNoQK97FXn7A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>> . >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "go-cd" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAFD%2B7DnZo77QzutmB1SwUHw%2B2sGC5JNjXVkWWyXGFzTn7OQdzg%40mail.gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAFD%2B7DnZo77QzutmB1SwUHw%2B2sGC5JNjXVkWWyXGFzTn7OQdzg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>> . >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>>> in the Google Groups "go-cd" group. >>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>> https://groups.google.com/d/topic/go-cd/wEK4-BYReb8/unsubscribe >>>>>>>>>>>> . >>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>> >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAPKX9jZas%2BPcdOZ855o1ZgpRoi0J0X4%3DnWMXC%3DBg9GjzAwwb-A%40mail.gmail.com >>>>>>>>>>>> >>>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAPKX9jZas%2BPcdOZ855o1ZgpRoi0J0X4%3DnWMXC%3DBg9GjzAwwb-A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "go-cd" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAFD%2B7Dn%2BDPphBOumaogyhuPSObpi4t1Ji2TgBFjTzEQ7M2qq6w%40mail.gmail.com >>>>>>>>>>> >>>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAFD%2B7Dn%2BDPphBOumaogyhuPSObpi4t1Ji2TgBFjTzEQ7M2qq6w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "go-cd" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/go-cd/wEK4-BYReb8/unsubscribe. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to [email protected]. >>>>>>>>>> >>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAPKX9jZNHv83JEWZbm9R5jNMR9_h1Gk365ou6w%3DfGe4e5uDXQg%40mail.gmail.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAPKX9jZNHv83JEWZbm9R5jNMR9_h1Gk365ou6w%3DfGe4e5uDXQg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "go-cd" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/go-cd/CAFD%2B7DkReyQMze3UpoxKYOrvXn%2B1WC6ax2z8U%2Bs5YBrxjSU0%3DA%40mail.gmail.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/go-cd/CAFD%2B7DkReyQMze3UpoxKYOrvXn%2B1WC6ax2z8U%2Bs5YBrxjSU0%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "go-cd" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/go-cd/30b62739-5858-491b-a4ef-6b353592265cn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/go-cd/30b62739-5858-491b-a4ef-6b353592265cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "go-cd" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/go-cd/wEK4-BYReb8/unsubscribe. >>>> >>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/CAPKX9ja1hJirt0E6XWKcxcqiJtwO%3DUyy1g%2BsiSYpKgfbjsgbBA%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/go-cd/CAPKX9ja1hJirt0E6XWKcxcqiJtwO%3DUyy1g%2BsiSYpKgfbjsgbBA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Hussein Kadiri >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "go-cd" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/go-cd/CAFD%2B7D%3DLmXbHJHFNqULqQCYkMVWu0GLvoKJmuGgMTPUXQccU6w%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/go-cd/CAFD%2B7D%3DLmXbHJHFNqULqQCYkMVWu0GLvoKJmuGgMTPUXQccU6w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/CAPKX9jYF7ffnvezTWff0xg9VSj519ea0Pk4vx2AeyydnDVgkyA%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/go-cd/CAPKX9jYF7ffnvezTWff0xg9VSj519ea0Pk4vx2AeyydnDVgkyA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/8094adb8-ef1c-453d-9409-01d08d3352bbn%40googlegroups.com.
