** Description changed:
+ [Impact]
+
+ Docker uses containerd under the hood. When containerd is upgraded it
+ stops and restarts its service; docker stops when containerd stops but
+ doesn’t restart. Particularly when doing unattended upgrades, an SRU
+ fix rolled out for containerd can result in unexpected and widespread
+ service outages for docker.
+
+ [Test Case]
+
+ $ sudo apt install docker.io
+ $ sudo systemctl start docker
+ $ systemctl status docker | grep Active
+ Active: active (running) since[...]
+ $ systemctl status containerd | grep Active
+ Active: active (running) since[...]
+
+ $ docker pull ubuntu/redis:latest
+ $ docker run -e REDIS_PASSWORD=1234 --network host \
+ --name test-redis -d ubuntu/redis:latest
+ $ telnet localhost 6379
+ $ docker container logs test-redis
+
+ $ sudo apt install --reinstall containerd
+ $ systemctl status containerd | grep Active
+ Active: active (running) since
+ $ systemctl status docker | grep Active
+ Active: inactive (dead) since [...]; 8s ago
+ $ docker container logs test-redis
+
+ [Where Problems Could Occur]
+
+ The challenge with this issue is addressing all important corner cases,
+ and as such the biggest risk is that we miss a corner case and fail to
+ keep the two services running when they should. Areas to watch will be
+ failures during start/stop/restart/upgrade type operations. Issues
+ during runtime are unlikely to relate to this change.
+
+ [Original Report]
+
hello we have had in several vms, the problem that after updating to "
containerd:amd64 1.3.3" the docker daemon is stopped and so are all
running containers
the vms run with ubuntu 18.04.1/18.04.2
journal log during the update:
```
-- Logs begin at Tue 2020-01-14 09:58:27 CET, end at Fri 2020-04-03 11:30:39
CEST. --
Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00"
level=info msg="level=info ts=2020-04-03T04:09:09.648358396Z caller=loki.go:55
container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f
msg=\"ignoring empty line\" line="
plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00"
level=info msg="level=info ts=2020-04-03T04:09:09.660781933Z caller=loki.go:55
container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f
msg=\"ignoring empty line\" line=\" \""
plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03
06:09:14+02:00","message":"interface enp0s3 works fine"}
Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03
06:09:14+02:00","message":"interface enp0s8 works fine"}
Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00"
level=info msg="level=info ts=2020-04-03T04:09:24.650166885Z caller=loki.go:55
container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f
msg=\"ignoring empty line\" line="
plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00"
level=info msg="level=info ts=2020-04-03T04:09:24.663239678Z caller=loki.go:55
container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f
msg=\"ignoring empty line\" line=\" \""
plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:31 server systemd[1]: Starting Daily apt upgrade and clean
activities...
Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00"
level=info msg="level=info ts=2020-04-03T04:09:39.64981528Z caller=loki.go:55
container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f
msg=\"ignoring empty line\" line="
plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00"
level=info msg="level=info ts=2020-04-03T04:09:39.665086076Z caller=loki.go:55
container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f
msg=\"ignoring empty line\" line=\" \""
plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:43 server systemd[1]: Stopping Docker Application Container
Engine...
Apr 03 06:09:43 server dockerd[1751]:
time="2020-04-03T06:09:43.571408106+02:00" level=info msg="Processing signal
'terminated'"
Apr 03 06:09:43 server dockerd[1751]:
time="2020-04-03T06:09:43.614841270+02:00" level=info msg="Daemon shutdown
complete"
Apr 03 06:09:43 server dockerd[1751]:
time="2020-04-03T06:09:43.614972021+02:00" level=info msg="stopping event
stream following graceful shutdown" error="context canceled"
module=libcontainerd namespace=moby
Apr 03 06:09:43 server systemd[1]: Stopped Docker Application Container
Engine.
Apr 03 06:09:43 server systemd[1]: Stopping containerd container runtime...
Apr 03 06:09:43 server systemd[1]: Closed Docker Socket for the API.
Apr 03 06:09:43 server containerd[1736]:
time="2020-04-03T06:09:43.896546147+02:00" level=info msg="Stop CRI service"
Apr 03 06:09:43 server containerd[1736]:
time="2020-04-03T06:09:43.897819071+02:00" level=info msg="Stop CRI service"
Apr 03 06:09:43 server containerd[1736]:
time="2020-04-03T06:09:43.899864471+02:00" level=info msg="Event monitor
stopped"
Apr 03 06:09:43 server containerd[1736]:
time="2020-04-03T06:09:43.900291027+02:00" level=info msg="Stream server
stopped"
Apr 03 06:09:43 server systemd[1]: Stopped containerd container runtime.
Apr 03 06:09:47 server systemd[1]: Reloading.
Apr 03 06:09:48 server systemd[1]: Reloading.
Apr 03 06:09:49 server systemd[1]: Reloading.
Apr 03 06:09:49 server systemd[1]: Starting containerd container runtime...
Apr 03 06:09:49 server containerd[1904]:
time="2020-04-03T06:09:49.951538613+02:00" level=info msg="starting containerd"
revision= version="1.3.3-0ubuntu1~18.04.2"
Apr 03 06:09:49 server systemd[1]: Started containerd container runtime.
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.021260455+02:00" level=info msg="loading plugin
\"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.021461398+02:00" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.022876671+02:00" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.022992123+02:00" level=warning msg="failed to load
plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.023036103+02:00" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.031683852+02:00" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.031948994+02:00" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.032592571+02:00" level=info msg="loading plugin
\"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.034455570+02:00" level=info msg="skip loading plugin
\"io.containerd.snapshotter.v1.zfs\"..." error="path
/var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem
to be used with the zfs snapshotter: skip plugin"
type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.034808794+02:00" level=info msg="loading plugin
\"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.034914325+02:00" level=warning msg="could not use
snapshotter devmapper in metadata plugin" error="devmapper not configured"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.034952775+02:00" level=info msg="metadata content
store policy set" policy=shared
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035299108+02:00" level=info msg="loading plugin
\"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035362929+02:00" level=info msg="loading plugin
\"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035585401+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.containers-service\"..."
type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035642662+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035676962+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035751653+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035797984+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035899825+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.namespaces-service\"..."
type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035955175+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.035994665+02:00" level=info msg="loading plugin
\"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.504599571+02:00" level=info msg="loading plugin
\"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.504847245+02:00" level=info msg="loading plugin
\"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.507240519+02:00" level=info msg="loading plugin
\"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.514716375+02:00" level=info msg="loading plugin
\"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515138699+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515385472+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515439083+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515474183+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515521113+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515569144+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515602784+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515645985+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515812346+02:00" level=info msg="loading plugin
\"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.515960658+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.516016449+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.516062909+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.516218490+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.516437722+02:00" level=info msg="Start cri plugin
with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs
DefaultRuntimeName:runc DefaultRuntime:{Type: Engine: PodAnnotations:[] Root:
Options:<nil> PrivilegedWithoutHostDevices:false}
UntrustedWorkloadRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil>
PrivilegedWithoutHostDevices:false}
Runtimes:map[runc:{Type:io.containerd.runc.v1 Engine: PodAnnotations:[] Root:
Options:<nil> PrivilegedWithoutHostDevices:false}] NoPivot:false}
CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d
NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate:}
Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}]
Configs:map[] Auths:map[]} DisableTCPService:true StreamServerAddress:127.0.0.1
StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false
SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false
EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:}
MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false
RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false}
ContainerdRootDir:/var/lib/containerd
ContainerdEndpoint:/run/containerd/containerd.sock
RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri
StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.516732916+02:00" level=info msg="Connect containerd
service"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.517528134+02:00" level=info msg="Get image filesystem
path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.517888717+02:00" level=error msg="Failed to load cni
during init, please check CRI plugin status before setting up network for pods"
error="cni config load failed: no network config found in /etc/cni/net.d: cni
plugin not initialized: failed to load cni config"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.518436663+02:00" level=info msg="loading plugin
\"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.518837507+02:00" level=info msg="Start subscribing
containerd event"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.519057900+02:00" level=info msg="Start recovering
state"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.519208531+02:00" level=info msg=serving...
address=/run/containerd/containerd.sock.ttrpc
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.519444914+02:00" level=info msg=serving...
address=/run/containerd/containerd.sock
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.519481864+02:00" level=info msg="Start event monitor"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.519573005+02:00" level=info msg="Start snapshots
syncer"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.519657075+02:00" level=info msg="Start streaming
server"
Apr 03 06:09:50 server containerd[1904]:
time="2020-04-03T06:09:50.519497564+02:00" level=info msg="containerd
successfully booted in 0.569453s"
Apr 03 06:10:08 server systemd[1]: Reloading.
Apr 03 06:10:08 server systemd[1]: Stopping LSB: automatic crash report
generation...
Apr 03 06:10:08 server apport[2377]: * Stopping automatic crash report
generation: apport
Apr 03 06:10:08 server apport[2377]: ...done.
Apr 03 06:10:08 server systemd[1]: Stopped LSB: automatic crash report
generation.
Apr 03 06:10:09 server systemd[1]: Reloading.
Apr 03 06:10:09 server systemd[1]: Reloading.
Apr 03 06:10:11 server systemd[1]: Reloading.
Apr 03 06:10:11 server systemd[1]: Reloading.
Apr 03 06:10:12 server systemd[1]: Reloading.
Apr 03 06:10:12 server systemd[1]: Starting LSB: automatic crash report
generation...
Apr 03 06:10:12 server apport[2666]: * Starting automatic crash report
generation: apport
Apr 03 06:10:12 server apport[2666]: ...done.
Apr 03 06:10:12 server systemd[1]: Started LSB: automatic crash report
generation.
Apr 03 06:10:12 server systemd[1]: Reloading.
Apr 03 06:10:18 server systemd[1]: Started Daily apt upgrade and clean
activities.
```
** Summary changed:
- update containerd:amd64 1.3.3-0 stops docker daemon
+ [SRU] update containerd:amd64 1.3.3-0 stops docker daemon
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1870514
Title:
[SRU] update containerd:amd64 1.3.3-0 stops docker daemon
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1870514/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
