Re: hRe: [go-cd] More complex git workflows

Is creating a new PV and provisioning that a feasible option? That's what I do on my Illumos build zones for some use cases - I use a ZFS snapshot, mount it into a Zone and run specific build experiments. It saves me a lot of time since I can avoid cloning gigabytes of code even from the same

Re: hRe: [go-cd] More complex git workflows

Thank you Hussein for that, glad to help. You may have seen this article here: https://medium.com/@Sushil_Kumar/readwritemany-persistent-volumes-in-google-kubernetes-engine-a0b93e203180 I think this person works around limitations on readwritemany by creating separate PVs fir read and write

Re: hRe: [go-cd] More complex git workflows

Thank you Marques, I really appreciate your help and support. You've been very responsive. I'm looking for an alternative to Jenkins. Unfortunately we have really large repos so most existing CI tools don't work out of the box because of the clone issues. I really like GoCD and have enjoyed

Re: hRe: [go-cd] More complex git workflows

Ah, I see. Probably would need some weird workaround to get this working then, which may not be worth your time/effort. I think for us to support —reference, it would be straightforward but not exactly trivial either. It would require a modest code change from what I recall since I was working in

Re: hRe: [go-cd] More complex git workflows

Ok, fair enough :). Let us know if we can help in any way. On Mon, May 10, 2021 at 7:06 PM HUSSEIN KADIRI wrote: > Let me explore the local clone option some more > $ git clone /local/path/to/bare/repo > > Thanks for the tip > On Monday, 10 May 2021 at 19:01:34 UTC-7 HUSSEIN KADIRI wrote: > >>

Re: hRe: [go-cd] More complex git workflows

Let me explore the local clone option some more $ git clone /local/path/to/bare/repo Thanks for the tip On Monday, 10 May 2021 at 19:01:34 UTC-7 HUSSEIN KADIRI wrote: > You're right the agent won't need read write. The volume is what needs to > be setup as read write. See doc >

hRe: [go-cd] More complex git workflows

You're right the agent won't need read write. The volume is what needs to be setup as read write. See doc Write - would be the "cron" updating the repo Read - would be the agents "cloning" Reading and Writing can't

Re: [go-cd] More complex git workflows

Typo — “why would the agents need read/write” was what I meant On Mon, May 10, 2021 at 6:53 PM Marques Lee wrote: > Pardon my lack of familiarity with GKE, my question may be extremely naive > - if there’s only process that needs to write to the mount (the cron that > performs the fetch to keep

Re: [go-cd] More complex git workflows

Pardon my lack of familiarity with GKE, my question may be extremely naive - if there’s only process that needs to write to the mount (the cron that performs the fetch to keep the code updated), then by would the agents need read/write on the volume? Isn’t read-only enough to clone for a pipeline

Re: [go-cd] More complex git workflows

Hmm, we setup GoCD on kubernetes so the server is a kubernetes deployment. We are on GKE . Yes we can mount a volume but GKE PVC are only ReadWriteOnce (One pod mounted to read and write) or ReadOnlyMany (Multiple pods mounted and reading only from the volume). Your recommendation requires a

Re: [go-cd] More complex git workflows

Basically the agents would be doing git clone /local/path/to/bare/repo In the scenario I was supposing On Mon, May 10, 2021 at 6:43 PM Marques Lee wrote: > Hmm, unfortunately I don’t believe the git material in GoCD has native > support for reference repos. > > Could the server not run a cron

Re: [go-cd] More complex git workflows

Hmm, unfortunately I don’t believe the git material in GoCD has native support for reference repos. Could the server not run a cron to do git fetch —all on the bare repo to keep it updated? Then all agents would pick up new changes so long as the volume holding the bare repo is mounted to

Re: [go-cd] More complex git workflows

The reference repo is a recent copy (1 day old) of the repo. The content would be used for the build. A build would clone from the reference repo, then fetch origin. This way the delta (1 day) is not too big. Cloning from scratch takes a long time because of the size of the repo. On Mon, May

Re: [go-cd] More complex git workflows

What does your reference repo contain? Is the content going to be used for the build? Ram On Tue, 11 May 2021 at 9:37 AM, HUSSEIN KADIRI wrote: > I have elastic agents so the cron route would not be feasible. > > I mount a reference repo as a K8s PVC. > > I want to do a git clone

Re: [go-cd] Re: Push based CI workflow pipeline material

Your pipeline does not need code? GoCD pipelines require some kind of material at least. Here a material need not be a code repo. It could be an upstream pipeline, or a package repository (eg, rpm) or a custom type based on a plugin. As a side note, a webhook from, say, GitHub merely triggers the

Re: [go-cd] Re: Push based CI workflow pipeline material

You could create a dummy material repo, and either trigger the pipeline via an API call or via a commit to that repo or via a schedule. Ram On Tue, 11 May 2021 at 9:30 AM, HUSSEIN KADIRI wrote: > Hmm, > Yes the web hook would make it push based. But It still does not get me > past the material

Re: [go-cd] More complex git workflows

I have elastic agents so the cron route would not be feasible. I mount a reference repo as a K8s PVC. I want to do a git clone --reference-repo . Can the git material support or be modified to accept reference repo paths? This is kind of a deal breaker if it can't On Mon, May 10, 2021, 6:31

Re: [go-cd] More complex git workflows

On Tue, 11 May 2021 at 9:31 AM, Marques Lee wrote: > Hmm. Well I suppose you could git clone —bare —mirror to a known location > and then: > > 1) create git materials with file system paths for the url instead of > http/ssh > 2) run a cron to keep the ref repo updated > > Would that work? >

Re: [go-cd] More complex git workflows

Hmm. Well I suppose you could git clone —bare —mirror to a known location and then: 1) create git materials with file system paths for the url instead of http/ssh 2) run a cron to keep the ref repo updated Would that work? On Mon, May 10, 2021 at 6:25 PM HUSSEIN KADIRI wrote: > My repo is too

[go-cd] Re: Push based CI workflow pipeline material

Hmm, Yes the web hook would make it push based. But It still does not get me past the material requirement. I don't want to configure a material as I have no need for it in my use case. I just want a pipeline that gets triggered from an external source. On Mon, May 10, 2021, 6:13 PM HUSSEIN

Re: [go-cd] More complex git workflows

My repo is too big, it can't be cloned by itself. it needs a reference repo. Is there a way to configure a reference repo ? On Mon, May 10, 2021, 6:23 PM Marques Lee wrote: > Git materials support shallow clone. I think you need to expand the > advance tab to see the option. It’s of course in

Re: [go-cd] More complex git workflows

Git materials support shallow clone. I think you need to expand the advance tab to see the option. It’s of course in cruise-config.xml as well as the various pipelines-as-config syntaxes. The workspace isn’t recloned every time either. If it exists on disk, it gets updated via fetch. On Mon, May

Re: [go-cd] Push based CI workflow pipeline material

https://api.gocd.org/current/#hosting-on-bitbucket If you keep scrolling it will show you the webhook setup for GitHub, etc. Side note:?If you are looking for webhook support in a PR workflow (where each PR generates its own pipeline), the Groovy DSL plugin has first class support for this. We

Re: [go-cd] Push based CI workflow pipeline material

GoCD does support webhook from GitHub gitlab, bitbucket.com and self hosted bitbucket. Let me find the docs for that... On Mon, May 10, 2021 at 6:14 PM HUSSEIN KADIRI wrote: > Hi again, > My CI workflow is push based (i.e some external tool makes an API call to > trigger a workflow). >

[go-cd] Push based CI workflow pipeline material

Hi again, My CI workflow is push based (i.e some external tool makes an API call to trigger a workflow). Materials in GoCD seem to imply a pull based system (i.e polling a repo, triggered from an upstream pipeline repo, etc) which is not my case. Being that materials are required fields and are

[go-cd] More complex git workflows

Hi, Sometimes a git repo is large and so it's not efficient/fast to do a git clone. One would have to use a reference repo. For all the parts that require git (config repository, git material, etc), is it possible to have more complex git workflows - utilizing a reference repo, shallow clone -

Re: [go-cd] API to trigger a pipeline

Ashwanth beat me to it. Also interesting may be the api to run an individual stage in a pipeline: https://api.gocd.org/current/#run-stage On Mon, May 10, 2021 at 4:22 PM 'Ashwanth Kumar' via go-cd < go-cd@googlegroups.com> wrote: > Check out the "Scheduling pipelines" section. > > On Tue, May

Re: [go-cd] API to trigger a pipeline

Check out the "Scheduling pipelines" section. On Tue, May 11, 2021, 4:49 AM HUSSEIN KADIRI wrote: > Hi, > My use case involves an external system triggering a CI workflow. > > I can trigger a pipeline by hitting the play button on the UI. I'm looking > thru the API docs for pipelines: >

[go-cd] API to trigger a pipeline

Hi, My use case involves an external system triggering a CI workflow. I can trigger a pipeline by hitting the play button on the UI. I'm looking thru the API docs for pipelines: https://api.gocd.org/current/#pipelines I'm not seeing a corresponding way of triggering a pipeline via an API. Could