That doesn't appear to be an error -- that's just stating that it found a dead client that was holding the exclusice-lock, so it broke the dead client's lock on the image (by blacklisting the client).
On Fri, Jan 25, 2019 at 5:09 AM ST Wong (ITSC) <[email protected]> wrote: > > Oops, while I can map and mount the filesystem, still found error as below, > while rebooting the client machine freezes and have to power reset her. > > Jan 25 17:57:30 acapp1 kernel: XFS (rbd0): Mounting V5 Filesystem > Jan 25 17:57:30 acapp1 kernel: rbd: rbd0: client74700 seems dead, breaking > lock ß > Jan 25 17:57:30 acapp1 kernel: XFS (rbd0): Starting recovery (logdev: > internal) > Jan 25 17:57:30 acapp1 kernel: XFS (rbd0): Ending recovery (logdev: internal) > Jan 25 17:58:07 acapp1 kernel: rbd: rbd1: capacity 10737418240 features 0x5 > Jan 25 17:58:14 acapp1 kernel: XFS (rbd1): Mounting V5 Filesystem > Jan 25 17:58:14 acapp1 kernel: rbd: rbd1: client74700 seems dead, breaking > lock ß > Jan 25 17:58:15 acapp1 kernel: XFS (rbd1): Starting recovery (logdev: > internal) > Jan 25 17:58:15 acapp1 kernel: XFS (rbd1): Ending recovery (logdev: internal) > > Would you help ? Thanks. > /st > > -----Original Message----- > From: ceph-users <[email protected]> On Behalf Of ST Wong > (ITSC) > Sent: Friday, January 25, 2019 5:58 PM > To: [email protected] > Cc: [email protected] > Subject: Re: [ceph-users] RBD client hangs > > Hi, It works. Thanks a lot. > > /st > > -----Original Message----- > From: Jason Dillaman <[email protected]> > Sent: Tuesday, January 22, 2019 9:29 PM > To: ST Wong (ITSC) <[email protected]> > Cc: Ilya Dryomov <[email protected]>; [email protected] > Subject: Re: [ceph-users] RBD client hangs > > Your "mon" cap should be "profile rbd" instead of "allow r" [1]. > > [1] > http://docs.ceph.com/docs/master/rbd/rados-rbd-cmds/#create-a-block-device-user > > On Mon, Jan 21, 2019 at 9:05 PM ST Wong (ITSC) <[email protected]> wrote: > > > > Hi, > > > > > Is this an upgraded or a fresh cluster? > > It's a fresh cluster. > > > > > Does client.acapp1 have the permission to blacklist other clients? You > > > can check with "ceph auth get client.acapp1". > > > > No, it's our first Ceph cluster with basic setup for testing, without any > > blacklist implemented. > > > > --------------- cut here ----------- > > # ceph auth get client.acapp1 > > exported keyring for client.acapp1 > > [client.acapp1] > > key = <key here> > > caps mds = "allow r" > > caps mgr = "allow r" > > caps mon = "allow r" > > caps osd = "allow rwx pool=2copy, allow rwx pool=4copy" > > --------------- cut here ----------- > > > > Thanks a lot. > > /st > > > > > > > > -----Original Message----- > > From: Ilya Dryomov <[email protected]> > > Sent: Monday, January 21, 2019 7:33 PM > > To: ST Wong (ITSC) <[email protected]> > > Cc: [email protected] > > Subject: Re: [ceph-users] RBD client hangs > > > > On Mon, Jan 21, 2019 at 11:43 AM ST Wong (ITSC) <[email protected]> > > wrote: > > > > > > Hi, we’re trying mimic on an VM farm. It consists 4 OSD hosts (8 OSDs) > > > and 3 MON. We tried mounting as RBD and CephFS (fuse and kernel > > > mount) on different clients without problem. > > > > Is this an upgraded or a fresh cluster? > > > > > > > > Then one day we perform failover test and stopped one of the OSD. Not > > > sure if it’s related but after that testing, the RBD client freeze when > > > trying to mount the rbd device. > > > > > > > > > > > > Steps to reproduce: > > > > > > > > > > > > # modprobe rbd > > > > > > > > > > > > (dmesg) > > > > > > [ 309.997587] Key type dns_resolver registered > > > > > > [ 310.043647] Key type ceph registered > > > > > > [ 310.044325] libceph: loaded (mon/osd proto 15/24) > > > > > > [ 310.054548] rbd: loaded > > > > > > > > > > > > # rbd -n client.acapp1 map 4copy/foo > > > > > > /dev/rbd0 > > > > > > > > > > > > # rbd showmapped > > > > > > id pool image snap device > > > > > > 0 4copy foo - /dev/rbd0 > > > > > > > > > > > > > > > > > > Then hangs if I tried to mount or reboot the server after rbd map. > > > There are lot of error in dmesg, e.g. > > > > > > > > > > > > Jan 20 03:43:32 acapp1 kernel: rbd: rbd0: blacklist of client74700 > > > failed: -13 > > > > > > Jan 20 03:43:32 acapp1 kernel: rbd: rbd0: failed to acquire lock: > > > -13 > > > > > > Jan 20 03:43:32 acapp1 kernel: rbd: rbd0: no lock owners detected > > > > > > Jan 20 03:43:32 acapp1 kernel: rbd: rbd0: client74700 seems dead, > > > breaking lock > > > > > > Jan 20 03:43:32 acapp1 kernel: rbd: rbd0: blacklist of client74700 > > > failed: -13 > > > > > > Jan 20 03:43:32 acapp1 kernel: rbd: rbd0: failed to acquire lock: > > > -13 > > > > > > Jan 20 03:43:32 acapp1 kernel: rbd: rbd0: no lock owners detected > > > > Does client.acapp1 have the permission to blacklist other clients? You can > > check with "ceph auth get client.acapp1". If not, follow step 6 of > > http://docs.ceph.com/docs/master/releases/luminous/#upgrade-from-jewel-or-kraken. > > > > Thanks, > > > > Ilya > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Jason > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Jason _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
