On Sunday, 8 October 2017 12:20:00 BST Hubert Hauser wrote: > On 07/10/17 18:50, Mick wrote: > > On Saturday, 7 October 2017 17:23:33 BST Hubert Hauser wrote: > >> I am using Gentoo as Host OS for Docker containers. > >> I have compiled > >> kernel using instructions on page > >> https://wiki.gentoo.org/wiki/Docker#Kernel and I have installed Docker > >> from Gentoo repository. > >> > >> Host system informations: > >> pecan@tux ~ $ uname -a > >> Linux tux 4.12.12-gentoo #8 SMP Sat Oct 7 13:58:47 CEST 2017 x86_64 > >> > >> Intel(R) Core(TM) i5-6300HQ CPU @ 2.30GHz GenuineIntel GNU/Linux > >> > >> Docker version: > >> pecan@tux ~ $ docker version > >> > >> Client: > >> Version: 17.03.2-ce > >> API version: 1.27 > >> Go version: go1.9.1 > >> Git commit: f5ec1e2 > >> Built: Sat Oct 7 14:50:59 2017 > >> OS/Arch: linux/amd64 > >> > >> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. > >> > >> Is the docker daemon running? > >> > >> Look at "Cannot connect to the Docker daemon at > >> unix:///var/run/docker.sock. Is the docker daemon running?". The same > >> > >> message appears if I try get docker system-wide informations: > >> pecan@tux ~ $ docker info > >> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. > >> > >> Is the docker daemon running? > > > > It seems you have not yet started docker. > > I can't start it because failure occurs when starting Docker daemon. > > On 07/10/17 18:50, Mick wrote: > >> The same error appears if I try run the same command as sudo, so this > >> error applies to daemon. I tried to check if there a mistake in Docker > >> daemon privileges. > >> > >> pecan@tux ~ $ sudo docker info > >> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. > >> > >> Is the docker daemon running? > >> > >> Based on the message I am able to say that maybe Docker daemon not > >> > >> running. I checked daemon status to make sure: > >> pecan@tux ~ $ sudo service docker status > >> > >> * status: crashed > > > > Did you try starting it from the CLI? Any useful messages there? > > Yes. > > The result of command sudo dockerd: > https://paste.pound-python.org/show/AyjLQEBGABpjo57E9WMa/ > and of sudo dockerd -s overlay2: > https://paste.pound-python.org/show/ulyqb7D363I51JqkvrEk/. > > You can also look at attachments dockerd.txt (sudo dockerd) and > dockerd-overlay2.txt (sudo dockerd -s overlay2). > > I must run dockerd without connecting into VPN. > > I've reset iptables and ip6tables rules to defaults: > > pecan@tux ~ $ sudo iptables -L > Chain INPUT (policy ACCEPT) > target prot opt source destination > > Chain FORWARD (policy ACCEPT) > target prot opt source destination > DOCKER-ISOLATION all -- anywhere anywhere > DOCKER all -- anywhere anywhere > ACCEPT all -- anywhere anywhere ctstate > RELATED,ESTABLISHED > ACCEPT all -- anywhere anywhere > ACCEPT all -- anywhere anywhere > > Chain OUTPUT (policy ACCEPT) > target prot opt source destination > > Chain DOCKER (1 references) > target prot opt source destination > > Chain DOCKER-ISOLATION (1 references) > target prot opt source destination > RETURN all -- anywhere anywhere > > (tensorflow) pecan@tux ~ $ sudo ip6tables -L > Chain INPUT (policy ACCEPT) > target prot opt source destination > > Chain FORWARD (policy ACCEPT) > target prot opt source destination > > Chain OUTPUT (policy ACCEPT) > target prot opt source destination > > Here's my Docker logs: > > (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker-err.log > time="2017-10-07T14:54:24.561887891+02:00" level=info > msg="libcontainerd: new containerd process, pid: 994" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-07T14:56:44.682987498+02:00" level=info > msg="libcontainerd: new containerd process, pid: 4983" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-07T17:00:16.938724808+02:00" level=info > msg="libcontainerd: new containerd process, pid: 7254" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-07T17:03:58.151814419+02:00" level=info > msg="libcontainerd: new containerd process, pid: 7857" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-07T17:05:09.492506131+02:00" level=info > msg="libcontainerd: new containerd process, pid: 7964" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-07T17:06:27.126674008+02:00" level=info > msg="libcontainerd: new containerd process, pid: 8202" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-07T17:09:47.482580356+02:00" level=info > msg="libcontainerd: new containerd process, pid: 8730" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-07T21:16:26.088461863+02:00" level=info > msg="libcontainerd: new containerd process, pid: 6414" > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-08T10:20:56.011379547+02:00" level=debug msg="docker group > found. gid: 987" > time="2017-10-08T10:20:56.023468730+02:00" level=debug msg="Listener > created for HTTP on unix (/var/run/docker.sock)" > time="2017-10-08T10:20:56.023686388+02:00" level=info > msg="libcontainerd: new containerd process, pid: 26284" > time="2017-10-08T10:20:56.030713286+02:00" level=debug msg="containerd: > read past events" count=0 > time="2017-10-08T10:20:56.030808207+02:00" level=debug msg="containerd: > supervisor running" cpus=4 memory=7849 runtime=docker-runc > runtimeArgs=[] stateDir="/var/run/docker/libcontainerd/containerd" > time="2017-10-08T10:20:56.030871453+02:00" level=debug msg="containerd: > grpc api on /var/run/docker/libcontainerd/docker-containerd.sock" > time="2017-10-08T10:20:56.524998596+02:00" level=debug > msg="libcontainerd: containerd health check returned error: rpc error: > code = 14 desc = grpc: the connection is unavailable" > time="2017-10-08T10:20:57.035916104+02:00" level=debug msg="Using > default logging driver json-file" > time="2017-10-08T10:20:57.035960712+02:00" level=debug msg="Golang's > threads limit set to 56430" > time="2017-10-08T10:20:57.036012700+02:00" level=debug > msg="[graphdriver] trying provided driver: btrfs" > time="2017-10-08T10:20:57.036027034+02:00" level=debug msg="Cleaning up > old mountid : start." > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > time="2017-10-08T10:21:05.597682241+02:00" level=debug msg="docker group > found. gid: 987" > time="2017-10-08T10:21:05.597724031+02:00" level=debug msg="Listener > created for HTTP on unix (/var/run/docker.sock)" > time="2017-10-08T10:21:05.597910790+02:00" level=info > msg="libcontainerd: new containerd process, pid: 26392" > time="2017-10-08T10:21:05.606126357+02:00" level=debug msg="containerd: > read past events" count=0 > time="2017-10-08T10:21:05.606219342+02:00" level=debug msg="containerd: > supervisor running" cpus=4 memory=7849 runtime=docker-runc > runtimeArgs=[] stateDir="/var/run/docker/libcontainerd/containerd" > time="2017-10-08T10:21:05.606245984+02:00" level=debug msg="containerd: > grpc api on /var/run/docker/libcontainerd/docker-containerd.sock" > time="2017-10-08T10:21:06.114930279+02:00" level=debug > msg="libcontainerd: containerd health check returned error: rpc error: > code = 14 desc = grpc: the connection is unavailable" > time="2017-10-08T10:21:06.614858989+02:00" level=debug > msg="libcontainerd: containerd health check returned error: rpc error: > code = 14 desc = grpc: the connection is unavailable" > time="2017-10-08T10:21:06.645208467+02:00" level=debug msg="Using > default logging driver json-file" > time="2017-10-08T10:21:06.645236784+02:00" level=debug msg="Golang's > threads limit set to 56430" > time="2017-10-08T10:21:06.645311511+02:00" level=debug > msg="[graphdriver] trying provided driver: btrfs" > time="2017-10-08T10:21:06.645324876+02:00" level=debug msg="Cleaning up > old mountid : start." > Error starting daemon: error initializing graphdriver: prerequisites for > driver not satisfied (wrong filesystem?) > (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker.log > time="2017-10-07T14:52:13.178261811+02:00" level=info > msg="libcontainerd: new containerd process, pid: 32311" > time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph > migration to content-addressability took 0.00 seconds" > time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your > kernel does not support cgroup blkio weight" > time="2017-10-07T14:52:14.434423960+02:00" level=warning msg="Your > kernel does not support cgroup blkio weight_device" > time="2017-10-07T14:52:14.434759986+02:00" level=info msg="Loading > containers: start." > time="2017-10-07T14:52:14.437180876+02:00" level=info msg="Firewalld > running: false" > Error starting daemon: Error initializing network controller: list > bridge addresses failed: no available network > (tensorflow) pecan@tux ~ $ sudo cat /var/log/docker-out.log > (tensorflow) pecan@tux ~ $ > > Docker daemon options: > > (tensorflow) pecan@tux ~ $ sudo cat /etc/docker/daemon.json > { > "debug": true > } > > On 07/10/17 18:50, Mick wrote: > >> Docker daemon is crashed. To see the reason, I looked at the logs: > >> pecan@tux ~ $ cat /var/log/docker.log > >> time="2017-10-07T14:52:13.178261811+02:00" level=info > >> > >> msg="libcontainerd: new containerd process, pid: 32311" > >> > >> time="2017-10-07T14:52:14.434232306+02:00" level=info msg="Graph > >> > >> migration to content-addressability took 0.00 seconds" > >> > >> time="2017-10-07T14:52:14.434413425+02:00" level=warning msg="Your > >> > >> kernel does not support cgroup blkio weight" > > > > OK, start from checking your kernel has all the necessary modules compiled > > in, rebuild it and reboot. > > Look here: > > pecan@tux ~ $ sudo ~/check-config.sh > info: reading kernel config from /proc/config.gz ... > > Generally Necessary: > - cgroup hierarchy: properly mounted [/sys/fs/cgroup] > - CONFIG_NAMESPACES: enabled > - CONFIG_NET_NS: enabled > - CONFIG_PID_NS: enabled > - CONFIG_IPC_NS: enabled > - CONFIG_UTS_NS: enabled > - CONFIG_CGROUPS: enabled > - CONFIG_CGROUP_CPUACCT: enabled > - CONFIG_CGROUP_DEVICE: enabled > - CONFIG_CGROUP_FREEZER: enabled > - CONFIG_CGROUP_SCHED: enabled > - CONFIG_CPUSETS: enabled > - CONFIG_MEMCG: enabled > - CONFIG_KEYS: enabled > - CONFIG_VETH: enabled > - CONFIG_BRIDGE: enabled > - CONFIG_BRIDGE_NETFILTER: enabled > - CONFIG_NF_NAT_IPV4: enabled > - CONFIG_IP_NF_FILTER: enabled > - CONFIG_IP_NF_TARGET_MASQUERADE: enabled > - CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled > - CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled > - CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module) > - CONFIG_IP_NF_NAT: enabled > - CONFIG_NF_NAT: enabled > - CONFIG_NF_NAT_NEEDED: enabled > - CONFIG_POSIX_MQUEUE: enabled > > Optional Features: > - CONFIG_USER_NS: enabled > - CONFIG_SECCOMP: enabled > - CONFIG_CGROUP_PIDS: enabled > - CONFIG_MEMCG_SWAP: enabled > - CONFIG_MEMCG_SWAP_ENABLED: enabled > (cgroup swap accounting is currently enabled) > - CONFIG_LEGACY_VSYSCALL_EMULATE: enabled > - CONFIG_BLK_CGROUP: enabled > - CONFIG_BLK_DEV_THROTTLING: enabled > - CONFIG_IOSCHED_CFQ: enabled (as module) > - CONFIG_CFQ_GROUP_IOSCHED: missing > - CONFIG_CGROUP_PERF: enabled > - CONFIG_CGROUP_HUGETLB: missing > - CONFIG_NET_CLS_CGROUP: enabled > - CONFIG_CGROUP_NET_PRIO: enabled > - CONFIG_CFS_BANDWIDTH: enabled > - CONFIG_FAIR_GROUP_SCHED: enabled > - CONFIG_RT_GROUP_SCHED: enabled > - CONFIG_IP_VS: enabled (as module) > - CONFIG_IP_VS_NFCT: enabled > - CONFIG_IP_VS_RR: enabled (as module) > - CONFIG_EXT3_FS: enabled > - CONFIG_EXT3_FS_XATTR: missing > - CONFIG_EXT3_FS_POSIX_ACL: enabled > - CONFIG_EXT3_FS_SECURITY: enabled > (enable these ext3 configs if you are using ext3 as backing filesystem) > - CONFIG_EXT4_FS: enabled > - CONFIG_EXT4_FS_POSIX_ACL: enabled > - CONFIG_EXT4_FS_SECURITY: enabled > - Network Drivers: > - "overlay": > - CONFIG_VXLAN: enabled (as module) > Optional (for encrypted networks): > - CONFIG_CRYPTO: enabled > - CONFIG_CRYPTO_AEAD: enabled > - CONFIG_CRYPTO_GCM: enabled > - CONFIG_CRYPTO_SEQIV: enabled > - CONFIG_CRYPTO_GHASH: enabled > - CONFIG_XFRM: enabled > - CONFIG_XFRM_USER: enabled (as module) > - CONFIG_XFRM_ALGO: enabled (as module) > - CONFIG_INET_ESP: enabled (as module) > - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled (as module) > - "ipvlan": > - CONFIG_IPVLAN: enabled (as module) > - "macvlan": > - CONFIG_MACVLAN: enabled (as module) > - CONFIG_DUMMY: enabled (as module) > - "ftp,tftp client in container": > - CONFIG_NF_NAT_FTP: enabled > - CONFIG_NF_CONNTRACK_FTP: enabled > - CONFIG_NF_NAT_TFTP: enabled > - CONFIG_NF_CONNTRACK_TFTP: enabled > - Storage Drivers: > - "aufs": > - CONFIG_AUFS_FS: missing > - "btrfs": > - CONFIG_BTRFS_FS: enabled > - CONFIG_BTRFS_FS_POSIX_ACL: enabled > - "devicemapper": > - CONFIG_BLK_DEV_DM: enabled > - CONFIG_DM_THIN_PROVISIONING: enabled > - "overlay": > - CONFIG_OVERLAY_FS: enabled > - "zfs": > - /dev/zfs: missing > - zfs command: missing > - zpool command: missing > > Limits: > - /proc/sys/kernel/keys/root_maxkeys: 1000000 > > My kernel config: > https://paste.pound-python.org/show/bFHNp4jrHE8kKzPMtNGq/ (in attachment > file kernel-config.txt, sudo zcat /proc/config.gz). > > -- > Best regards, > Hubert Hauser.
I have not trimmed your info above, in case someone more knowledgeable in running docker can advise. >From what I see above you are running btrfs. It may be worth compiling in your kernel this module you have left out, because I've read somewhere it prevents fs corruption (Rich seems to know a lot about BTRFS, so I leave it to him to confirm its usefulness): # CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set Two warnings in /var/log/docker-err.log attracted my attention: "Your kernel does not support cgroup blkio weight" Try settings fair queue scheduling for your storage and check it is being used. The second thing is caused by the filesystem: ... prerequisites for driver not satisfied (wrong filesystem?) You could try starting docker with options '-d -g' which apparently works for booting btrfs as advised here: https://stackoverflow.com/questions/30669359/docker-with-btrfs-ubuntu HTH. -- Regards, Mick
signature.asc
Description: This is a digitally signed message part.
