Hello,

I'm running GoCD: 24.3.0 with an elastic agent profile running on AWS EKS
cluster. My pipelines run properly most of the time, but sometimes certain
runs get into some limbo state with pod logs as below. When it does, the
pipeline is just stuck waiting on assigning agents.

Is there anything obvious for anyone who has seen this error before? Should
I just upgrade to 24.5.0 and see? The only way I get out of this is by
terminating my node on EC2 forcefully. Killing the pod also doesn't help,
because all newer pods have the same issue.

$ sudo /run-docker-daemon.sh
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
/run-docker-daemon.sh: line 23:     9 Killed                  $(which dind)
dockerd --host=unix:///var/run/docker.sock
${DOCKERD_ADDITIONAL_ARGS:-'--host=tcp://localhost:2375'} >
/var/log/dockerd.log 2>&1
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
dockerd startup failed...
time="2024-12-23T14:35:58.923528879Z" level=info msg="Starting up"
time="2024-12-23T14:35:58.924393339Z" level=warning msg="Binding to IP
address without --tlsverify is insecure and gives root access on this
machine to everyone who has access to your network."
host="tcp://localhost:2375"
time="2024-12-23T14:35:58.924414660Z" level=warning msg="Binding to an IP
address, even on localhost, can also give access to scripts run in a
browser. Be safe out there!" host="tcp://localhost:2375"
time="2024-12-23T14:35:58.924442010Z" level=warning msg="[DEPRECATION
NOTICE] In future versions this will be a hard failure preventing the
daemon from starting! Learn more at:
https://docs.docker.com/go/api-security/"; host="tcp://localhost:2375"
time="2024-12-23T14:35:59.925995985Z" level=info msg="containerd not
running, starting managed containerd"
time="2024-12-23T14:35:59.928570903Z" level=info msg="started new
containerd process" address=/var/run/docker/containerd/containerd.sock
module=libcontainerd pid=141
time="2024-12-23T14:35:59.956936540Z" level=info msg="starting containerd"
revision=8fc6bcff51318944179630522a095cc9dbf9f353 version=v1.7.20
time="2024-12-23T14:36:00.000700964Z" level=info msg="loading plugin
\"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2024-12-23T14:36:00.000778006Z" level=info msg="loading plugin
\"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2024-12-23T14:36:00.001422560Z" level=info msg="loading plugin
\"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1
time="2024-12-23T14:36:00.001454791Z" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.blockfile\"..."
type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.001564513Z" level=info msg="skip loading plugin
\"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file
generator: skip plugin" type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.001590374Z" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.devmapper\"..."
type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.001613715Z" level=info msg="skip loading plugin
\"io.containerd.snapshotter.v1.devmapper\"..." error="devmapper not
configured: skip plugin" type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.001631625Z" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.native\"..."
type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.001791238Z" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.overlayfs\"..."
type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.002375252Z" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.018509514Z" level=info msg="skip loading plugin
\"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported
(modprobe aufs failed: exit status 1 \"ip: can't find device
'aufs'\\nmodprobe: can't change directory to '/lib/modules': No such file
or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.018690338Z" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.019063176Z" level=info msg="skip loading plugin
\"io.containerd.snapshotter.v1.zfs\"..." error="path
/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be
a zfs filesystem to be used with the zfs snapshotter: skip plugin"
type=io.containerd.snapshotter.v1
time="2024-12-23T14:36:00.019096327Z" level=info msg="loading plugin
\"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2024-12-23T14:36:00.019363883Z" level=info msg="loading plugin
\"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2024-12-23T14:36:00.019483946Z" level=info msg="metadata content
store policy set" policy=shared
time="2024-12-23T14:36:00.025371189Z" level=info msg="loading plugin
\"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2024-12-23T14:36:00.025466381Z" level=info msg="loading plugin
\"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2024-12-23T14:36:00.025513112Z" level=info msg="loading plugin
\"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
time="2024-12-23T14:36:00.025568054Z" level=info msg="loading plugin
\"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1
time="2024-12-23T14:36:00.025598394Z" level=info msg="loading plugin
\"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2024-12-23T14:36:00.025975922Z" level=info msg="loading plugin
\"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2024-12-23T14:36:00.026418103Z" level=info msg="loading plugin
\"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2024-12-23T14:36:00.026879433Z" level=info msg="loading plugin
\"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
time="2024-12-23T14:36:00.026911204Z" level=info msg="loading plugin
\"io.containerd.sandbox.store.v1.local\"..."
type=io.containerd.sandbox.store.v1
time="2024-12-23T14:36:00.026934074Z" level=info msg="loading plugin
\"io.containerd.sandbox.controller.v1.local\"..."
type=io.containerd.sandbox.controller.v1
time="2024-12-23T14:36:00.026959075Z" level=info msg="loading plugin
\"io.containerd.service.v1.containers-service\"..."
type=io.containerd.service.v1
time="2024-12-23T14:36:00.026981415Z" level=info msg="loading plugin
\"io.containerd.service.v1.content-service\"..."
type=io.containerd.service.v1
time="2024-12-23T14:36:00.027001746Z" level=info msg="loading plugin
\"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2024-12-23T14:36:00.027134208Z" level=info msg="loading plugin
\"io.containerd.service.v1.images-service\"..."
type=io.containerd.service.v1
time="2024-12-23T14:36:00.027172850Z" level=info msg="loading plugin
\"io.containerd.service.v1.introspection-service\"..."
type=io.containerd.service.v1
time="2024-12-23T14:36:00.027207840Z" level=info msg="loading plugin
\"io.containerd.service.v1.namespaces-service\"..."
type=io.containerd.service.v1
time="2024-12-23T14:36:00.027228851Z" level=info msg="loading plugin
\"io.containerd.service.v1.snapshots-service\"..."
type=io.containerd.service.v1
time="2024-12-23T14:36:00.027250042Z" level=info msg="loading plugin
\"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2024-12-23T14:36:00.027280312Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027302052Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027322173Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027343804Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027364884Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027385374Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027486097Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027533828Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027563219Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027596579Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027616510Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027644821Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027668330Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027701042Z" level=info msg="loading plugin
\"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
time="2024-12-23T14:36:00.027736933Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027756793Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.027778393Z" level=info msg="loading plugin
\"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2024-12-23T14:36:00.027988048Z" level=info msg="loading plugin
\"io.containerd.tracing.processor.v1.otlp\"..."
type=io.containerd.tracing.processor.v1
time="2024-12-23T14:36:00.028029879Z" level=info msg="skip loading plugin
\"io.containerd.tracing.processor.v1.otlp\"..." error="skip plugin: tracing
endpoint not configured" type=io.containerd.tracing.processor.v1
time="2024-12-23T14:36:00.028055519Z" level=info msg="loading plugin
\"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2024-12-23T14:36:00.028086000Z" level=info msg="skip loading plugin
\"io.containerd.internal.v1.tracing\"..." error="skip plugin: tracing
endpoint not configured" type=io.containerd.internal.v1
time="2024-12-23T14:36:00.028110501Z" level=info msg="loading plugin
\"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2024-12-23T14:36:00.028163852Z" level=info msg="loading plugin
\"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
time="2024-12-23T14:36:00.028196603Z" level=info msg="NRI interface is
disabled by configuration."
time="2024-12-23T14:36:00.028682933Z" level=info msg=serving...
address=/var/run/docker/containerd/containerd-debug.sock
time="2024-12-23T14:36:00.028929999Z" level=info msg=serving...
address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2024-12-23T14:36:00.029152474Z" level=info msg=serving...
address=/var/run/docker/containerd/containerd.sock
time="2024-12-23T14:36:00.029203025Z" level=info msg="containerd
successfully booted in 0.073659s"
time="2024-12-23T14:36:01.003481197Z" level=info msg="Loading containers:
start."
time="2024-12-23T14:36:02.301133968Z" level=info msg="Default bridge
(docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip
can be used to set a preferred IP address"
time="2024-12-23T14:36:02.600676172Z" level=info msg="Loading containers:
done."
time="2024-12-23T14:36:02.635377282Z" level=warning msg="[DEPRECATION
NOTICE]: API is accessible on http://localhost:2375 without encryption.\n
      Access to the remote API is equivalent to root access on the host.
Refer\n         to the 'Docker daemon attack surface' section in the
documentation for\n         more information:
https://docs.docker.com/go/attack-surface/\nIn future versions this will be
a hard failure preventing the daemon from starting! Learn more at:
https://docs.docker.com/go/api-security/";
time="2024-12-23T14:36:02.635459914Z" level=info msg="Docker daemon"
commit=cc13f95 containerd-snapshotter=false storage-driver=overlay2
version=27.1.1
time="2024-12-23T14:36:02.635735160Z" level=info msg="Daemon has completed
initialization"
/docker-entrypoint.sh: cannot sudo /run-docker-daemon.sh

-- 

Ashwanth Kumar / ashwanthkumar.in

-- 
You received this message because you are subscribed to the Google Groups 
"go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to go-cd+unsubscr...@googlegroups.com.
To view this discussion visit 
https://groups.google.com/d/msgid/go-cd/CAD9m7Cz8R3sZtuNL_T49g_SW%2BqjdGjt087c-6NrbbiqSgh4QYg%40mail.gmail.com.

Reply via email to