Hello, I'm running GoCD: 24.3.0 with an elastic agent profile running on AWS EKS cluster. My pipelines run properly most of the time, but sometimes certain runs get into some limbo state with pod logs as below. When it does, the pipeline is just stuck waiting on assigning agents.
Is there anything obvious for anyone who has seen this error before? Should I just upgrade to 24.5.0 and see? The only way I get out of this is by terminating my node on EC2 forcefully. Killing the pod also doesn't help, because all newer pods have the same issue. $ sudo /run-docker-daemon.sh Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? /run-docker-daemon.sh: line 23: 9 Killed $(which dind) dockerd --host=unix:///var/run/docker.sock ${DOCKERD_ADDITIONAL_ARGS:-'--host=tcp://localhost:2375'} > /var/log/dockerd.log 2>&1 Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? dockerd startup failed... time="2024-12-23T14:35:58.923528879Z" level=info msg="Starting up" time="2024-12-23T14:35:58.924393339Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://localhost:2375" time="2024-12-23T14:35:58.924414660Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://localhost:2375" time="2024-12-23T14:35:58.924442010Z" level=warning msg="[DEPRECATION NOTICE] In future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/" host="tcp://localhost:2375" time="2024-12-23T14:35:59.925995985Z" level=info msg="containerd not running, starting managed containerd" time="2024-12-23T14:35:59.928570903Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=141 time="2024-12-23T14:35:59.956936540Z" level=info msg="starting containerd" revision=8fc6bcff51318944179630522a095cc9dbf9f353 version=v1.7.20 time="2024-12-23T14:36:00.000700964Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1 time="2024-12-23T14:36:00.000778006Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1 time="2024-12-23T14:36:00.001422560Z" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1 time="2024-12-23T14:36:00.001454791Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.001564513Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.001590374Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.001613715Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." error="devmapper not configured: skip plugin" type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.001631625Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.001791238Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.002375252Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.018509514Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"ip: can't find device 'aufs'\\nmodprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.018690338Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.019063176Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1 time="2024-12-23T14:36:00.019096327Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1 time="2024-12-23T14:36:00.019363883Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1 time="2024-12-23T14:36:00.019483946Z" level=info msg="metadata content store policy set" policy=shared time="2024-12-23T14:36:00.025371189Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1 time="2024-12-23T14:36:00.025466381Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1 time="2024-12-23T14:36:00.025513112Z" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1 time="2024-12-23T14:36:00.025568054Z" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1 time="2024-12-23T14:36:00.025598394Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1 time="2024-12-23T14:36:00.025975922Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1 time="2024-12-23T14:36:00.026418103Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2 time="2024-12-23T14:36:00.026879433Z" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2 time="2024-12-23T14:36:00.026911204Z" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1 time="2024-12-23T14:36:00.026934074Z" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1 time="2024-12-23T14:36:00.026959075Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.026981415Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.027001746Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.027134208Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.027172850Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.027207840Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.027228851Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.027250042Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1 time="2024-12-23T14:36:00.027280312Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027302052Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027322173Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027343804Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027364884Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027385374Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027486097Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027533828Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027563219Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027596579Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027616510Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027644821Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027668330Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027701042Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1 time="2024-12-23T14:36:00.027736933Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027756793Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.027778393Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1 time="2024-12-23T14:36:00.027988048Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1 time="2024-12-23T14:36:00.028029879Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.tracing.processor.v1 time="2024-12-23T14:36:00.028055519Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1 time="2024-12-23T14:36:00.028086000Z" level=info msg="skip loading plugin \"io.containerd.internal.v1.tracing\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.internal.v1 time="2024-12-23T14:36:00.028110501Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1 time="2024-12-23T14:36:00.028163852Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1 time="2024-12-23T14:36:00.028196603Z" level=info msg="NRI interface is disabled by configuration." time="2024-12-23T14:36:00.028682933Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock time="2024-12-23T14:36:00.028929999Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc time="2024-12-23T14:36:00.029152474Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock time="2024-12-23T14:36:00.029203025Z" level=info msg="containerd successfully booted in 0.073659s" time="2024-12-23T14:36:01.003481197Z" level=info msg="Loading containers: start." time="2024-12-23T14:36:02.301133968Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" time="2024-12-23T14:36:02.600676172Z" level=info msg="Loading containers: done." time="2024-12-23T14:36:02.635377282Z" level=warning msg="[DEPRECATION NOTICE]: API is accessible on http://localhost:2375 without encryption.\n Access to the remote API is equivalent to root access on the host. Refer\n to the 'Docker daemon attack surface' section in the documentation for\n more information: https://docs.docker.com/go/attack-surface/\nIn future versions this will be a hard failure preventing the daemon from starting! Learn more at: https://docs.docker.com/go/api-security/" time="2024-12-23T14:36:02.635459914Z" level=info msg="Docker daemon" commit=cc13f95 containerd-snapshotter=false storage-driver=overlay2 version=27.1.1 time="2024-12-23T14:36:02.635735160Z" level=info msg="Daemon has completed initialization" /docker-entrypoint.sh: cannot sudo /run-docker-daemon.sh -- Ashwanth Kumar / ashwanthkumar.in -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/go-cd/CAD9m7Cz8R3sZtuNL_T49g_SW%2BqjdGjt087c-6NrbbiqSgh4QYg%40mail.gmail.com.