Hi On Mon, Jan 8, 2018 at 9:27 PM, Jason Dillaman <[email protected]> wrote:
> If you are using a pre-created RBD image for this, you will need to > disable all the image features that krbd doesn't support: > > # rbd feature disable dummy01 exclusive-lock,object-map, > fast-diff,deep-flatten > > I've tried this with a pre-created image. Unfortunately no luck. When I try creating a volume directly with rbd-docker-plugin, the attempt hangs indefinitely, with no feedback I can use to debug further: --- root@lol-server2:~# /usr/local/bin/rbd-docker-plugin --create -debug -name dummy01 -size 1024 -cluster ceph -user ceph -fs xfs -config /etc/ceph/ceph.conf 2018/01/13 09:49:57 main.go:92: INFO: starting rbd-docker-plugin version 1.5.0 2018/01/13 09:49:57 main.go:93: INFO: canCreateVolumes=%!q(bool=true), removeAction="ignore" 2018/01/13 09:49:57 main.go:103: INFO: Setting up Ceph Driver for PluginID=dummy01, cluster=ceph, user=ceph, pool=rbd, mount=/var/lib/docker-volumes, config=/etc/ceph/ceph.conf, go-ceph=%!s(bool=false) 2018/01/13 09:49:57 driver.go:115: INFO: newCephRBDVolumeDriver: setting base mount dir=/var/lib/docker-volumes/dummy01 2018/01/13 09:49:57 main.go:127: INFO: Creating Docker VolumeDriver Handler 2018/01/13 09:49:57 main.go:131: INFO: Opening Socket for Docker to connect: /run/docker/plugins/dummy01.sock --- Is there any way I can diagnose at a lower level to understand what's going on here? > On Sun, Jan 7, 2018 at 11:36 AM, Traiano Welcome <[email protected]> > wrote: > > Hi List > > > > I'm getting the following error when trying to run docker with a rbd > volume > > (either pre-existing, or not): > > > > "VolumeDriver.Create: Unable to create Ceph RBD Image" > > > > Please could someone give me a clue as to how to debug this further and > > resolve it? > > > > Details of my platform: > > > > 1. ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe) > > 2. Docker version 17.05.0-ce, build 89658be > > 3. rbd-docker-plugin --version 2.0.1 > > 4. Kernel: Linux lol-server-049 4.4.0-62-generic #83-Ubuntu SMP Wed Jan > 18 > > 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux > > > > Here are the details from the rbd-docker logs and syslogs: > > > > - Running docker with an as-yet-uncreated rbd volume, and > rbd-docker-plugin > > with --create=true: > > > > ``` > > root@lol-server-045:~# docker run --volume-driver=rbd --volume > dummy02:/mnt > > centos:latest bash > > docker: Error response from daemon: create dummy02: VolumeDriver.Create: > > Unable to create Ceph RBD Image(dummy02): exit status 2. > > See 'docker run --help'. > > ``` > > > > - With an already created rbd volume, and rbd-docker-plugin with > > --create=false: > > > > ``` > > root@lol-server-045:~# docker run --volume-driver=rbd --volume > dummy01:/mnt > > centos:latest bash > > docker: Error response from daemon: create dummy01: VolumeDriver.Create: > > Ceph RBD Image not found: dummy01. > > > > ``` > > > > > > - state of a pre-created rbd device: > > > > root@lol-server-045:/var/log# rbd ls| egrep dummy > > dummy01 > > > > root@lol-server-045:/var/log# rbd info dummy01 > > rbd image 'dummy01': > > size 1096 MB in 274 objects > > order 22 (4096 kB objects) > > block_name_prefix: rbd_data.85d6238e1f29 > > format: 2 > > features: layering, exclusive-lock, object-map, fast-diff, > > deep-flatten > > flags: > > > > BUG: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1578484 > > > > ``` > > root@lol-server-045:/var/log# > > root@lol-server-045:/var/log# rbd feature disable foo exclusive-lock > > object-map fast-diff deep-flatten > > rbd: error opening image foo: (2) No such file or directory > > root@lol-server-045:/var/log# rbd feature disable dummy01 exclusive-lock > > object-map fast-diff deep-flatten > > root@lol-server-045:/var/log# rbd map dummy01 --pool rbd > > /dev/rbd3 > > ``` > > > > - rbd-docker-plugin.log entry following restart of the rbd-docker driver > > service) > > > > ``` > > 2018/01/07 23:45:20 main.go:121: INFO: Creating Docker VolumeDriver > Handler > > 2018/01/07 23:45:20 main.go:125: INFO: Opening Socket for Docker to > connect: > > /run/docker/plugins/rbd.sock > > 2018/01/07 23:45:29 main.go:141: INFO: received TERM or KILL signal: > > terminated > > 2018/01/07 23:45:29 main.go:190: INFO: closing log file > > 2018/01/07 23:45:29 main.go:91: INFO: starting rbd-docker-plugin version > > 2.0.1 > > 2018/01/07 23:45:29 main.go:92: INFO: canCreateVolumes=true, > > removeAction="ignore" > > 2018/01/07 23:45:29 main.go:101: INFO: Setting up Ceph Driver for > > PluginID=rbd, cluster=, ceph-user=docker, pool=rbd, > > mount=/var/lib/docker-volumes, config=/etc/ceph/ceph.conf > > 2018/01/07 23:45:29 driver.go:85: INFO: newCephRBDVolumeDriver: setting > base > > mount dir=/var/lib/docker-volumes/rbd > > 2018/01/07 23:45:29 main.go:121: INFO: Creating Docker VolumeDriver > Handler > > 2018/01/07 23:45:29 main.go:125: INFO: Opening Socket for Docker to > connect: > > /run/docker/plugins/rbd.sock > > ``` > > > > - when attempting to run a docker image, specifying a volume that does > not > > yet exist: > > > > ``` > > root@lol-server-045:/var/log# docker run -u 0 --privileged -it > > --volume-driver rbd -v dummy02:/mnt:rw centos:latest bash > > ``` > > > > > > docker: Error response from daemon: create dummy02: VolumeDriver.Create: > > Unable to create Ceph RBD Image(dummy02): exit status 2. > > ``` > > > > - Log entry: > > > > ``` > > 2018/01/07 23:45:29 driver.go:85: INFO: newCephRBDVolumeDriver: setting > base > > mount dir=/var/lib/docker-volumes/rbd > > 2018/01/07 23:45:29 main.go:121: INFO: Creating Docker VolumeDriver > Handler > > 2018/01/07 23:45:29 main.go:125: INFO: Opening Socket for Docker to > connect: > > /run/docker/plugins/rbd.sock > > 2018/01/07 23:46:56 api.go:188: Entering go-plugins-helpers getPath > > 2018/01/07 23:46:56 driver.go:467: WARN: Image dummy02 does not exist > > 2018/01/07 23:46:56 api.go:132: Entering go-plugins-helpers createPath > > 2018/01/07 23:46:56 driver.go:145: INFO: API Create(&{"dummy02" map[]}) > > 2018/01/07 23:46:56 driver.go:153: INFO: createImage(&{"dummy02" map[]}) > > 2018/01/07 23:46:56 driver.go:687: INFO: Attempting to create new RBD > Image: > > (rbd/dummy02, %!s(int=20480), xfs) > > 2018/01/07 23:46:56 driver.go:203: ERROR: Unable to create Ceph RBD > > Image(dummy02): exit status 2 > > ``` > > > > - docker log entries: > > > > > > ``` > > Jan 7 23:42:03 lol-server-045 kernel: [4063726.059726] aufs > > au_opts_verify:1597:dockerd[107149]: dirperm1 breaks the protection by > the > > permission bits on the lower branch > > Jan 7 23:42:30 lol-server-045 kernel: [4063752.624828] aufs > > au_opts_verify:1597:dockerd[107147]: dirperm1 breaks the protection by > the > > permission bits on the lower branch > > Jan 7 23:45:20 lol-server-045 rbd-docker-plugin[77813]: 2018/01/07 > 23:45:20 > > main.go:179: INFO: setting log file: /var/log/rbd-docker-plugin.log > > Jan 7 23:45:29 lol-server-045 rbd-docker-plugin[77856]: 2018/01/07 > 23:45:29 > > main.go:179: INFO: setting log file: /var/log/rbd-docker-plugin.log > > Jan 7 23:46:56 lol-server-045 kernel: [4064019.169722] aufs > > au_opts_verify:1597:dockerd[107449]: dirperm1 breaks the protection by > the > > permission bits on the lower branch > > Jan 7 23:46:56 lol-server-045 dockerd[107120]: > > time="2018-01-07T23:46:56.857163090+08:00" level=error msg="Handler for > POST > > /v1.29/containers/create returned error: create dummy02: > > VolumeDriver.Create: Unable to create Ceph RBD Image(dummy02): exit > status > > 2" > > ``` > > > > - state of the ceph cluster: > > > > ``` > > root@lol-server-045:/var/log# ceph -s > > cluster 0bb54801-846d-47ac-b14a-3828d830ff3a > > health HEALTH_OK > > monmap e1: 1 mons at {lol-server-045=10.0.0.20:6789/0} > > election epoch 6, quorum 0 lol-server-045 > > fsmap e11: 1/1/1 up {0=lol-server-050=up:active} > > osdmap e64: 5 osds: 5 up, 5 in > > flags sortbitwise,require_jewel_osds > > pgmap v1232770: 192 pgs, 3 pools, 14067 MB data, 82167 objects > > > > 2018/01/07 23:45:29 main.go:125: INFO: Opening Socket for Docker to > connect: > > /run/docker/plugins/rbd.sock > > 2018/01/07 23:46:56 api.go:188: Entering go-plugins-helpers getPath > > 2018/01/07 23:46:56 driver.go:467: WARN: Image dummy02 does not exist > > 2018/01/07 23:46:56 api.go:132: Entering go-plugins-helpers createPath > > 2018/01/07 23:46:56 driver.go:145: INFO: API Create(&{"dummy02" map[]}) > > 2018/01/07 23:46:56 driver.go:153: INFO: createImage(&{"dummy02" map[]}) > > 2018/01/07 23:46:56 driver.go:687: INFO: Attempting to create new RBD > Image: > > (rbd/dummy02, %!s(int=20480), xfs) > > 2018/01/07 23:46:56 driver.go:203: ERROR: Unable to create Ceph RBD > > Image(dummy02): exit status 2 > > ``` > > > > - docker log entries: > > > > > > ``` > > Jan 7 23:42:03 lol-server-045 kernel: [4063726.059726] aufs > > au_opts_verify:1597:dockerd[107149]: dirperm1 breaks the protection by > the > > permission bits on the lower branch > > Jan 7 23:42:30 lol-server-045 kernel: [4063752.624828] aufs > > au_opts_verify:1597:dockerd[107147]: dirperm1 breaks the protection by > the > > permission bits on the lower branch > > Jan 7 23:45:20 lol-server-045 rbd-docker-plugin[77813]: 2018/01/07 > 23:45:20 > > main.go:179: INFO: setting log file: /var/log/rbd-docker-plugin.log > > Jan 7 23:45:29 lol-server-045 rbd-docker-plugin[77856]: 2018/01/07 > 23:45:29 > > main.go:179: INFO: setting log file: /var/log/rbd-docker-plugin.log > > Jan 7 23:46:56 lol-server-045 kernel: [4064019.169722] aufs > > au_opts_verify:1597:dockerd[107449]: dirperm1 breaks the protection by > the > > permission bits on the lower branch > > Jan 7 23:46:56 lol-server-045 dockerd[107120]: > > time="2018-01-07T23:46:56.857163090+08:00" level=error msg="Handler for > POST > > /v1.29/containers/create returned error: create dummy02: > > VolumeDriver.Create: Unable to create Ceph RBD Image(dummy02): exit > status > > 2" > > ``` > > > > - state of the ceph cluster: > > > > ``` > > root@lol-server-045:/var/log# ceph -s > > cluster 0bb54801-846d-47ac-b14a-3828d830ff3a > > health HEALTH_OK > > monmap e1: 1 mons at {lol-server-045=10.0.0.20:6789/0} > > election epoch 6, quorum 0 lol-server-045 > > fsmap e11: 1/1/1 up {0=lol-server-050=up:active} > > osdmap e64: 5 osds: 5 up, 5 in > > flags sortbitwise,require_jewel_osds > > pgmap v1232770: 192 pgs, 3 pools, 14067 MB data, 82167 objects > > 28396 MB used, 7623 GB / 7651 GB avail > > 192 active+clean > > ``` > > > > Many thanks in advance for any help! > > > > Traiano > > > > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > -- > Jason >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
