Hi Paul Thank you for suggestion. Firstly, a correction “I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1” is incorrect. I meant that version 12.2.8-0ubuntu0.18.04.1 on node N01 works, . 12.2.12-0ubuntu0.18.04.2 is having problem.
After upgrading to version 12.2.12-0ubuntu0.18.04.2, it doesn’t work anymore. No changes on client keys file or on the Ceph cluster. Therefore I deduce that it is a problem with client (ceph-common) software. ~ Vang From: Paul Emmerich <[email protected]> Date: Friday, 6 September 2019 at 13.13 To: Vang Le-Quy <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: [ceph-users] Ceph client failed to mount RBD device after reboot failure to break locks is usually due to incorrect permissions on the key, check if the client key has "profile rbd" mon permissions -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io<http://www.croit.io> Tel: +49 89 1896585 90 On Fri, Sep 6, 2019 at 1:06 PM Vang Le-Quy <[email protected]<mailto:[email protected]>> wrote: I am suspecting that there is a bug in 12.2.8-0ubuntu0.18.04.1. We have too ceph client n01 and n03 with following software versions. # n01 ceph-common: Installed: 12.2.8-0ubuntu0.18.04.1 Candidate: 12.2.12-0ubuntu0.18.04.2 # n03 ceph-common: Installed: 12.2.11-0ubuntu0.18.04.2 Candidate: 12.2.12-0ubuntu0.18.04.2 # n01 uname -a Linux n01 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # n03 uname -a Linux n03 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Problem: N01 can mount its RBD after reboot whereas N03 can’t. To verify that it is something on client side, I upgrade n01 to have the same package version to n03 including ceph-common. And n01 does indeed fail to mount its CEPH RBD partition. Syslog shows messages like the following: Sep 6 12:52:33 n01 kernel: [ 875.160275] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.161069] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.161723] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.161724] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.161956] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.162755] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.163415] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.163645] rbd: rbd0: no lock owners detected Sep 6 12:52:33 n01 kernel: [ 875.164431] rbd: rbd0: client2508785 seems dead, breaking lock Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: blacklist of client2508785 failed: -13 Sep 6 12:52:33 n01 kernel: [ 875.165082] rbd: rbd0: failed to acquire lock: -13 Sep 6 12:52:33 n01 kernel: [ 875.165355] rbd: rbd0: no lock owners detected [signature_1482564062] Vang Quy Le Special Consultant in Data Science and Infrastructure T: (+45) 9940 7710 | Email: [email protected]<mailto:[email protected]> Kontor 0-1-91 | Selma Lagerløfs Vej 300 | DK-9220 Aalborg Ø | _______________________________________________ ceph-users mailing list -- [email protected]<mailto:[email protected]> To unsubscribe send an email to [email protected]<mailto:[email protected]>
_______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
