Thank you for the quick reply, Chad. As suggested, I will try building an image with Buildah image as FROM and add all the Go agent dependencies. If not, I will try exploring some other tool instead of Buildah to build, push the image.
Regards, Raghu On Fri, Sep 20, 2024 at 12:16 PM Chad Wilson <ch...@thoughtworks.com> wrote: > Getting alternate container tools like buldah to work inside a container > with fuse overlays etc is a bit out of GoCD's scope right now. > > You should be able to attach volumes like for any other pod, but a host > path mount is obviously coupled to the way your nodes are configured within > Kubernetes and even further outside GoCD's control. > > Having said this, fuse is special I think? Normally there are *many* > things needed to make buildah work with fuse overlays inside a container > re: userns remappings to avoid enabling special privileges and linux > capabilities within securityContext which may not be there by default . You > *might* just be missing CAP_SETUID and CAP_SETGID capabilities from the > container and/or running with *privileged: true* but I'd be surprised if > that is all that is needed. > > > https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container#running_buildah_inside_a_container > https://github.com/containers/buildah/issues/2325 > https://github.com/containers/image_build/blob/main/buildah/README.md > https://github.com/containers/buildah/discussions/5218 > > As you can see from > https://github.com/containers/image_build/blob/main/buildah/Containerfile > it is totally non-trivial to handle the "general" case ... but you could > always try and copy and paste the same into your dockerfile and see where > you end up :p. > > While I would be very interested in this easier to get working by default > and have the GoCD container images prepare themself for this to make it > easier (or have a special image similar to the "dind" images), I have not > found a way that is sufficiently agnostic from host/node configurations > that is suitable for something like GoCD. I probably haven't looked hard > enough though. > > Anyway, to put this another way, this sounds really like a question of > "how do I run buildah within a Kubernetes pod with/without special > privileges?" rather than anything GoCD-specific. If GoCD can easily provide > an opinionated container agent image that supports this, and/or opinionated > configuration templates for pods in Helm charts and elastic agents then I > am keen for some help to find the right way, but not something I have tried > properly myself. > > Your alternative is to try changing your base image to one based on a > buildah image like quay.io/buldah/stable:latest, and then either > > - layer on GoCD's stuff in your custom image with > https://github.com/gocd/docker-gocd-agent-almalinux-9/blob/main/Dockerfile > (suggesting Alma base image, as think buildah images are fedora based) > *OR* > - avoid having to update the instructions to match GoCD changes every > release by using a multi-stage Dockerfile to pull across all of /go > /go-agent /godata /gocd-jre /docker-entrypoint.sh /docker-entrypoint.d > /usr/local/sbin/tini (and the UID/GID, ENV etc). This set of dirs isn't > considered a "stable API", but will probably be easier to maintain than > copy and pasting the raw Dockerfile instructions. > > e.g something like the below (untested!!!!) > > FROM quay.io/buldah/stable:latest > > # Install gocd-agent in container-ready form > ARG GO_AGENT_IMAGE=gocd/gocd-agent-almalinux-9 > ARG GO_VERSION=v24.3.0 > RUN useradd -l -u 1000 -g root -d /home/go -m go && \ > dnf install -y git-core openssh-clients bash unzip curl-minimal procps-ng > coreutils-single glibc-langpack-en tar && \ > dnf clean all && \ > rm -rf /var/cache/dnf && \ > COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /usr/local/sbin/tini > /usr/local/sbin/tini > COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /gocd-jre /gocd-jre > COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /go-agent /go-agent > COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /go /go > COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /godata /godata > COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /docker-entrypoint.d > /docker-entrypoint.d > COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /docker-entrypoint.sh > /docker-entrypoint.sh > ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8 > ENV GO_JAVA_HOME="/gocd-jre" > ENTRYPOINT ["/docker-entrypoint.sh"] > USER go > > > # Install your stuff? > > > You'll have to be careful with > > - all of the permissions though to make sure they come across with the > COPY instructions etc > - figure out if there is stuff in the buildah container entrypoints > that needs to come across > - validate that the buildah image is intended to use UID=1000 by > default or already has a user configured etc. > > > -Chad > > On Fri, Sep 20, 2024 at 1:56 PM Raghu Kumar <raghusgro...@gmail.com> > wrote: > >> Hello Chad, >> >> Continuing where Sloka left off. >> We have been able to build the GoCD agent and it does run the SBT >> commands. However, while trying to build an image using Buildah we are >> seeing certain errors like this: >> --- >> Error during unshare(CLONE_NEWUSER): No space left on device >> User namespaces are not enabled in /proc/sys/user/max_user_namespaces. >> time="2024-09-16T07:06:05Z" level=error msg="error parsing PID \"\": >> strconv.Atoi: parsing \"\": invalid syntax" >> time="2024-09-16T07:06:05Z" level=error msg="(unable to determine exit >> status)" >> Error during unshare(CLONE_NEWUSER): No space left on device >> User namespaces are not enabled in /proc/sys/user/max_user_namespaces. >> time="2024-09-16T07:06:05Z" level=error msg="error parsing PID \"\": >> strconv.Atoi: parsing \"\": invalid syntax" >> time="2024-09-16T07:06:05Z" level=error msg="(unable to determine exit >> status)" >> --- >> >> This makes me believe that there are certain storage constraints on the >> Elastic agent container. However, if I look at the elastic agent Pod >> configuration, I don't see any volume being attached. The pod configuration >> is mentioned below: >> --- >> apiVersion: v1 >> kind: Pod >> metadata: >> name: gocd-agent-{{ POD_POSTFIX }} >> labels: >> app: web >> spec: >> serviceAccountName: default >> containers: >> - name: gocd-agent-container-{{ CONTAINER_POSTFIX }} >> image: >> 366611831214.dkr.ecr.us-east-1.amazonaws.com/gocd/agent:ea-sbt-jdk11-build-2-1 >> volumeMounts: >> - name: ssh-secrets >> readOnly: true >> mountPath: /home/go/.ssh >> - name: dev-fuse >> mountPath: /dev/fuse >> env: >> - name: _BUILDAH_STARTED_IN_USERNS >> value: "" >> - name: STORAGE_DRIVER >> value: "overlay" >> - name: STORAGE_OPTS >> value: "overlay.mount_program=/usr/bin/fuse-overlayfs" >> securityContext: >> privileged: true >> capabilities: >> add: >> - SYS_ADMIN >> resources: >> limits: >> memory: "8192M" >> cpu: "2" >> requests: >> memory: "8192M" >> cpu: "2" >> volumes: >> - name: ssh-secrets >> secret: >> defaultMode: 420 >> secretName: gocd-bitbucket-kube-secret >> - name: dev-fuse >> hostPath: >> path: /dev/fuse >> type: CharDevice >> ... >> >> In case, I want to attach a volume to these agents how do I go about >> doing it? If that's not an option then do i need to increase the memory to >> accomodate larger materials for build? >> >> Thank you, >> Raghu >> On Wednesday 11 September 2024 at 13:35:56 UTC+5:30 Chad Wilson wrote: >> >>> You don't appear to have actually installed the GoCD agent in your image >>> - it needs to be there by default. You are better to base your container >>> image off one of the existing GoCD agent images from >>> https://www.gocd.org/download/#docker rather than trying to >>> hand-construct your own from scratch. There are Ubuntu variants available >>> similar to your current base e.g >>> https://hub.docker.com/r/gocd/gocd-agent-ubuntu-24.04 or >>> https://hub.docker.com/r/gocd/gocd-agent-ubuntu-22.04 >>> >>> -Chad >>> >>> On Wed, Sep 11, 2024 at 3:54 PM Sloka Roy <slokar...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I am trying to create GoCD custom elastic agent which supports Buildah >>>> for building docker images and pushing to ECR and SBT and JAVA 11 for >>>> compilation. >>>> >>>> Below dockerfile I plan to use as GoCD elastic agent. >>>> >>>> FROM eclipse-temurin:11.0.24_8-jdk-jammy # Install required packages >>>> including Buildah dependencies RUN apt-get update && \ apt-get install >>>> -y \ curl \ git \ zip \ unzip \ jq \ buildah \ runc \ fuse-overlayfs \ >>>> iptables >>>> && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Install sbt ARG >>>> SBT_VERSION=1.5.6 ENV SBT_HOME /usr/local/sbt ENV PATH >>>> ${PATH}:${SBT_HOME}/bin RUN curl -sL " >>>> https://github.com/sbt/sbt/releases/download/v1.5.6/sbt-1.5.6.tgz"; | >>>> gunzip | tar -x -C /usr/local && \ echo -ne "- with sbt >>>> $SBT_VERSION\n" >> /root/.built # Setup GoCD user and environment ENV >>>> HOME /var/go RUN groupadd -g 496 go && \ useradd -c "go user" -d $HOME >>>> -m go -g 496 -u 498 VOLUME /var/go WORKDIR /var/go USER go >>>> However the agent is not getting registered. >>>> Events: Type Reason Age From Message ---- ------ ---- ---- ------- >>>> Normal Scheduled 2m15s default-scheduler Successfully assigned >>>> gocd/k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 to >>>> ip-10-75-110-207.ec2.internal Normal Pulled 33s (x5 over 2m15s) kubelet >>>> Container image " >>>> 366611831214.dkr.ecr.us-east-1.amazonaws.com/gocd/agent:es-jdk11-build" >>>> already present on machine Normal Created 33s (x5 over 2m15s) kubelet >>>> Created container k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 Normal >>>> Started 32s (x5 over 2m15s) kubelet Started container >>>> k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 Warning BackOff 3s (x10 over >>>> 2m8s) kubelet Back-off restarting failed container >>>> k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 in pod >>>> k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017_gocd(8fe96d7b-ea06-4f80-a17b-13042f59c548) >>>> >>>> [image: Screenshot 2024-09-11 at 12.39.49 PM.png] >>>> >>>> Can you please help me here, with what are the minimum requirements to >>>> create an custom elastic agent >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "go-cd" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to go-cd+un...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/go-cd/cd320acc-2836-4d87-9a30-fa1d7233ce94n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/go-cd/cd320acc-2836-4d87-9a30-fa1d7233ce94n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "go-cd" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to go-cd+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/go-cd/5c17788e-422f-4f32-b5e3-bb709c73a3a9n%40googlegroups.com >> <https://groups.google.com/d/msgid/go-cd/5c17788e-422f-4f32-b5e3-bb709c73a3a9n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "go-cd" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to go-cd+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/go-cd/CAA1RwH8mB9xGwhGiJZPmq4e0_NQCwqpRdjBC-iTGe-sa3OFJKw%40mail.gmail.com > <https://groups.google.com/d/msgid/go-cd/CAA1RwH8mB9xGwhGiJZPmq4e0_NQCwqpRdjBC-iTGe-sa3OFJKw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAEafxuN01K8KSQP62kRuK67sSVs%3Dj7%2BoNQTQQay3GoU22Ys5pA%40mail.gmail.com.
