Has anyone run into this issue and would like to provide any troubleshooting tip?
Thanks, Jimmy From: J L <j...@yahoo-inc.com<mailto:j...@yahoo-inc.com>> Date: Thursday, June 5, 2014 at 4:20 PM To: "ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>" <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>> Subject: [ceph-users] OSD keyrings shifted and down Hello Ceph Guru, I rebooted osd server to fix “osd.33”. When the server came back online, I noticed all the osd are down, while I am troubleshooting and restarting the osd, I got below error for authentication. I also noticed the “keyring” for each osd had shifted. For example, for osd.33 which mapped to /var/lib/ceph/osd/ceph-33, its keyring should be mapped to [osd.33], in this case it mapped to [osd.34]. Can I just simply change the osd.# in the keyring to correct the mapping or is there proper for the fix? Please help. Thanks in advance!! -Jimmy [root@gfsnode1 ceph-34]# service ceph start osd.34 === osd.34 === 2014-06-05 15:08:54.053958 7f08f2b47700 0 librados: osd.34 authentication error (1) Operation not permitted Error connecting to cluster: PermissionError failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.34 --keyring=/var/lib/ceph/osd/ceph-34/keyring osd crush create-or-move -- 34 2.73 host=gfsnode1 root=default' [root@gfsnode1 ceph-34]# [root@gfsnode1 osd]# ls -l total 0 lrwxrwxrwx 1 root root 12 May 15 15:21 ceph-33 -> /ceph/osd120 lrwxrwxrwx 1 root root 12 May 15 15:22 ceph-34 -> /ceph/osd121 lrwxrwxrwx 1 root root 12 May 15 15:23 ceph-35 -> /ceph/osd122 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-36 -> /ceph/osd123 lrwxrwxrwx 1 root root 12 May 15 15:24 ceph-37 -> /ceph/osd124 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-38 -> /ceph/osd125 lrwxrwxrwx 1 root root 12 May 15 15:25 ceph-39 -> /ceph/osd126 lrwxrwxrwx 1 root root 12 May 15 15:26 ceph-40 -> /ceph/osd127 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-41 -> /ceph/osd128 lrwxrwxrwx 1 root root 12 May 15 15:27 ceph-42 -> /ceph/osd129 lrwxrwxrwx 1 root root 12 May 15 15:28 ceph-43 -> /ceph/osd130 [root@gfsnode1 osd]# cat ceph-33/keyring [osd.34] key = AQAwPnVT6G7fBRAA86D4FuxN0U8uKXk0brPbCQ== [root@gfsnode1 osd]# cat ceph-34/keyring [osd.35] key = AQBbPnVTmG4BLxAA6UV6XHbZepXUEXB6VJQzEA== [root@gfsnode1 osd]# cat ceph-35/keyring [osd.36] key = AQCDPnVTuL97JRAA1soDHToJ1c6WhXX+mnnRPw== [root@gfsnode1 osd]# cat ceph-36/keyring [osd.37] key = AQCwPnVTYAttNhAAomeRalOEHWlyO7C9tF+7SQ== [root@gfsnode1 osd]# cat ceph-37/keyring [osd.38] key = AQDKPnVTQC1DLBAAl0959S0st+UcFw8uOppa7g== [root@gfsnode1 osd]# cat ceph-38/keyring [osd.39] key = AQDjPnVTMFGwNxAABH5M1Y8uXoqecPesS09IGw== [root@gfsnode1 osd]# cat ceph-39/keyring [osd.40] key = AQChQXVT6JHiBxAAohTnBGxb2ZAbgCjt5M0xBw== [root@gfsnode1 osd]# cat ceph-40/keyring [osd.41] key = AQBGP3VTAHI0CRAAZkcUPLOFT1jx9v3DVNX4nQ== [root@gfsnode1 osd]# cat ceph-41/keyring [osd.42] key = AQAEsIdTMBTjChAAfJrsqIEBcCGEXv0jcK2vtQ== [root@gfsnode1 osd]# cat ceph-42/keyring [osd.43] key = AQB6P3VT2KW7ORAAU+1Ix/fUXIBU8jky0BQ9jw== [root@gfsnode1 osd]# cat ceph-43/keyring cat: ceph-43/keyring: No such file or directory [root@gfsnode1 osd]#
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
