Re: [ceph-users] rbd.ReadOnlyImage: [errno 30]

[解决]

Hello CUZA,
Are you rbd/disk_test in same ceph cluster? 
you export rbd/disk_test with one user while import rbd/disk_test with 
another one?






At 2019-06-05 23:25:45, "CUZA Frédéric"  wrote:
>Thank you all for you quick answer.
>I think that will solve our problem.
>
>This is what we came up with this :
>rbd -c /etc/ceph/Oceph.conf --keyring /etc/ceph/Oceph.client.admin.keyring 
>export rbd/disk_test - | rbd -c /etc/ceph/Nceph.conf --keyring 
>/etc/ceph/Nceph.client.admin.keyring import - rbd/disk_test
>
>This rbd image is a test with only 5Gb of datas inside of it.
>
>Unfortunately the command seems to be stuck and nothing happens, both ports 
>7800 / 6789 / 22.
>
>We can't find no logs on any monitors.
>
>Thanks !
>
>-Message d'origine-
>De : ceph-users  De la part de Jason 
>Dillaman
>Envoyé : 04 June 2019 14:14
>À : 解决 
>Cc : ceph-users 
>Objet : Re: [ceph-users] rbd.ReadOnlyImage: [errno 30]
>
>On Tue, Jun 4, 2019 at 4:55 AM 解决  wrote:
>>
>> Hi all,
>> We use ceph(luminous) + openstack(queens) in my test 
>> environment。The virtual machine does not start properly after the 
>> disaster test and the image of virtual machine can not create snap.The 
>> procedure is as follows：
>> #!/usr/bin/env python
>>
>> import rados
>> import rbd
>> with rados.Rados(conffile='/etc/ceph/ceph.conf',rados_id='nova') as cluster:
>> with cluster.open_ioctx('vms') as ioctx:
>> rbd_inst = rbd.RBD()
>> print "start open rbd image"
>> with rbd.Image(ioctx, '10df4634-4401-45ca-9c57-f349b78da475_disk') 
>> as image:
>> print "start create snapshot"
>> image.create_snap('myimage_snap1')
>>
>> when i run it ,it show readonlyimage,as follows:
>>
>> start open rbd image
>> start create snapshot
>> Traceback (most recent call last):
>>   File "testpool.py", line 17, in 
>> image.create_snap('myimage_snap1')
>>   File "rbd.pyx", line 1790, in rbd.Image.create_snap 
>> (/builddir/build/BUILD/ceph-12.2.5/build/src/pybind/rbd/pyrex/rbd.c:15
>> 682)
>> rbd.ReadOnlyImage: [errno 30] error creating snapshot myimage_snap1 
>> from 10df4634-4401-45ca-9c57-f349b78da475_disk
>>
>> but i run it with admin instead of nova,it is ok.
>>
>> "ceph auth list"  as follow
>>
>> installed auth entries:
>>
>> osd.1
>> key: AQBL7uRcfuyxEBAAoK8JrQWMU6EEf/g83zKJjg==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.10
>> key: AQCV7uRcdsB9IBAAHbHHCaylVUZIPKFX20polQ==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.11
>> key: AQCW7uRcRIMRIhAAbXfLbQwijEO5ZQFWFZaO5w==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.2
>> key: AQBL7uRcfFMWDBAAo7kjQobGBbIHYfZkx45pOw==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.4
>> key: AQBk7uRc97CPOBAAK9IBJICvchZPc5p80bISsg==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.5
>> key: AQBk7uRcOdqaORAAkQeEtYsE6rLWLPhYuCTdHA==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.7
>> key: AQB97uRc+1eRJxAA34DImQIMFjzHSXZ25djp0Q==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.8
>> key: AQB97uRcFilBJhAAXzSzNJsgwpobC8654Xo7Sw==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> client.admin
>> key: AQAU7uRcNia+BBAA09mOYdX+yJWbLCjcuMih0A==
>> auid: 0
>> caps: [mds] allow
>> caps: [mgr] allow *
>> caps: [mon] allow *
>> caps: [osd] allow *
>> client.cinder
>> key: AQBp7+RcOzPHGxAA7azgyayVu2RRNWJ7JxSJEg==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=volumes, allow rwx pool=volumes-cache, allow rwx pool=vms, allow 
>> rwx pool=vms-cache, allow rx pool=images, allow rx pool=images-cache 
>> client.cinder-backup
>> key: AQBq7+RcVOwGNRAAiwJ59ZvAUc0H4QkVeN82vA==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=backups, allow rwx pool=backups-cache client.glance
>> key: AQDf7uRc32hDBBAAkGucQEVTWqnIpNvihXf/Ng==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=images, allow rwx pool=images-cache client.nova
>> key: AQDN7+RcqDABIxAAXnFcVjBp/S5GkgOy0wqB1Q==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=volumes, allow rwx pool=volumes-cache,

Re: [ceph-users] rbd.ReadOnlyImage: [errno 30]

[解决]

Hello CUZA,
Are you rbd/disk_test in same ceph cluster? 
you export rbd/disk_test with one user while import rbd/disk_test with 
another one?


At 2019-06-05 23:25:45, "CUZA Frédéric"  wrote:
>Thank you all for you quick answer.
>I think that will solve our problem.
>
>This is what we came up with this :
>rbd -c /etc/ceph/Oceph.conf --keyring /etc/ceph/Oceph.client.admin.keyring 
>export rbd/disk_test - | rbd -c /etc/ceph/Nceph.conf --keyring 
>/etc/ceph/Nceph.client.admin.keyring import - rbd/disk_test
>
>This rbd image is a test with only 5Gb of datas inside of it.
>
>Unfortunately the command seems to be stuck and nothing happens, both ports 
>7800 / 6789 / 22.
>
>We can't find no logs on any monitors.
>
>Thanks !
>
>-Message d'origine-
>De : ceph-users  De la part de Jason 
>Dillaman
>Envoyé : 04 June 2019 14:14
>À : 解决 
>Cc : ceph-users 
>Objet : Re: [ceph-users] rbd.ReadOnlyImage: [errno 30]
>
>On Tue, Jun 4, 2019 at 4:55 AM 解决  wrote:
>>
>> Hi all,
>> We use ceph(luminous) + openstack(queens) in my test 
>> environment。The virtual machine does not start properly after the 
>> disaster test and the image of virtual machine can not create snap.The 
>> procedure is as follows：
>> #!/usr/bin/env python
>>
>> import rados
>> import rbd
>> with rados.Rados(conffile='/etc/ceph/ceph.conf',rados_id='nova') as cluster:
>> with cluster.open_ioctx('vms') as ioctx:
>> rbd_inst = rbd.RBD()
>> print "start open rbd image"
>> with rbd.Image(ioctx, '10df4634-4401-45ca-9c57-f349b78da475_disk') 
>> as image:
>> print "start create snapshot"
>> image.create_snap('myimage_snap1')
>>
>> when i run it ,it show readonlyimage,as follows:
>>
>> start open rbd image
>> start create snapshot
>> Traceback (most recent call last):
>>   File "testpool.py", line 17, in 
>> image.create_snap('myimage_snap1')
>>   File "rbd.pyx", line 1790, in rbd.Image.create_snap 
>> (/builddir/build/BUILD/ceph-12.2.5/build/src/pybind/rbd/pyrex/rbd.c:15
>> 682)
>> rbd.ReadOnlyImage: [errno 30] error creating snapshot myimage_snap1 
>> from 10df4634-4401-45ca-9c57-f349b78da475_disk
>>
>> but i run it with admin instead of nova,it is ok.
>>
>> "ceph auth list"  as follow
>>
>> installed auth entries:
>>
>> osd.1
>> key: AQBL7uRcfuyxEBAAoK8JrQWMU6EEf/g83zKJjg==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.10
>> key: AQCV7uRcdsB9IBAAHbHHCaylVUZIPKFX20polQ==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.11
>> key: AQCW7uRcRIMRIhAAbXfLbQwijEO5ZQFWFZaO5w==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.2
>> key: AQBL7uRcfFMWDBAAo7kjQobGBbIHYfZkx45pOw==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.4
>> key: AQBk7uRc97CPOBAAK9IBJICvchZPc5p80bISsg==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.5
>> key: AQBk7uRcOdqaORAAkQeEtYsE6rLWLPhYuCTdHA==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.7
>> key: AQB97uRc+1eRJxAA34DImQIMFjzHSXZ25djp0Q==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.8
>> key: AQB97uRcFilBJhAAXzSzNJsgwpobC8654Xo7Sw==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> client.admin
>> key: AQAU7uRcNia+BBAA09mOYdX+yJWbLCjcuMih0A==
>> auid: 0
>> caps: [mds] allow
>> caps: [mgr] allow *
>> caps: [mon] allow *
>> caps: [osd] allow *
>> client.cinder
>> key: AQBp7+RcOzPHGxAA7azgyayVu2RRNWJ7JxSJEg==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=volumes, allow rwx pool=volumes-cache, allow rwx pool=vms, allow 
>> rwx pool=vms-cache, allow rx pool=images, allow rx pool=images-cache 
>> client.cinder-backup
>> key: AQBq7+RcVOwGNRAAiwJ59ZvAUc0H4QkVeN82vA==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=backups, allow rwx pool=backups-cache client.glance
>> key: AQDf7uRc32hDBBAAkGucQEVTWqnIpNvihXf/Ng==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=images, allow rwx pool=images-cache client.nova
>> key: AQDN7+RcqDABIxAAXnFcVjBp/S5GkgOy0wqB1Q==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=volumes, allow rwx pool=volumes-cache,

Re: [ceph-users] rbd.ReadOnlyImage: [errno 30]

[解决]

Thank your help, jason
I find the reason.The exclusive-lock of image do not release after the 
disaster test. I release the exclusive-lock then The virtual machine start 
properly,and it can also create snap with nova user.


At 2019-06-04 20:13:35, "Jason Dillaman"  wrote:
>On Tue, Jun 4, 2019 at 4:55 AM 解决  wrote:
>>
>> Hi all,
>> We use ceph(luminous) + openstack(queens) in my test environment。The 
>> virtual machine does not start properly after the disaster test and the 
>> image of virtual machine can not create snap.The procedure is as follows：
>> #!/usr/bin/env python
>>
>> import rados
>> import rbd
>> with rados.Rados(conffile='/etc/ceph/ceph.conf',rados_id='nova') as cluster:
>> with cluster.open_ioctx('vms') as ioctx:
>> rbd_inst = rbd.RBD()
>> print "start open rbd image"
>> with rbd.Image(ioctx, '10df4634-4401-45ca-9c57-f349b78da475_disk') 
>> as image:
>> print "start create snapshot"
>> image.create_snap('myimage_snap1')
>>
>> when i run it ,it show readonlyimage,as follows:
>>
>> start open rbd image
>> start create snapshot
>> Traceback (most recent call last):
>>   File "testpool.py", line 17, in 
>> image.create_snap('myimage_snap1')
>>   File "rbd.pyx", line 1790, in rbd.Image.create_snap 
>> (/builddir/build/BUILD/ceph-12.2.5/build/src/pybind/rbd/pyrex/rbd.c:15682)
>> rbd.ReadOnlyImage: [errno 30] error creating snapshot myimage_snap1 from 
>> 10df4634-4401-45ca-9c57-f349b78da475_disk
>>
>> but i run it with admin instead of nova,it is ok.
>>
>> "ceph auth list"  as follow
>>
>> installed auth entries:
>>
>> osd.1
>> key: AQBL7uRcfuyxEBAAoK8JrQWMU6EEf/g83zKJjg==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.10
>> key: AQCV7uRcdsB9IBAAHbHHCaylVUZIPKFX20polQ==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.11
>> key: AQCW7uRcRIMRIhAAbXfLbQwijEO5ZQFWFZaO5w==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.2
>> key: AQBL7uRcfFMWDBAAo7kjQobGBbIHYfZkx45pOw==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.4
>> key: AQBk7uRc97CPOBAAK9IBJICvchZPc5p80bISsg==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.5
>> key: AQBk7uRcOdqaORAAkQeEtYsE6rLWLPhYuCTdHA==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.7
>> key: AQB97uRc+1eRJxAA34DImQIMFjzHSXZ25djp0Q==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> osd.8
>> key: AQB97uRcFilBJhAAXzSzNJsgwpobC8654Xo7Sw==
>> caps: [mon] allow profile osd
>> caps: [osd] allow *
>> client.admin
>> key: AQAU7uRcNia+BBAA09mOYdX+yJWbLCjcuMih0A==
>> auid: 0
>> caps: [mds] allow
>> caps: [mgr] allow *
>> caps: [mon] allow *
>> caps: [osd] allow *
>> client.cinder
>> key: AQBp7+RcOzPHGxAA7azgyayVu2RRNWJ7JxSJEg==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=volumes, allow rwx pool=volumes-cache, allow rwx pool=vms, allow rwx 
>> pool=vms-cache, allow rx pool=images, allow rx pool=images-cache
>> client.cinder-backup
>> key: AQBq7+RcVOwGNRAAiwJ59ZvAUc0H4QkVeN82vA==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=backups, allow rwx pool=backups-cache
>> client.glance
>> key: AQDf7uRc32hDBBAAkGucQEVTWqnIpNvihXf/Ng==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=images, allow rwx pool=images-cache
>> client.nova
>> key: AQDN7+RcqDABIxAAXnFcVjBp/S5GkgOy0wqB1Q==
>> caps: [mon] allow r
>> caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
>> pool=volumes, allow rwx pool=volumes-cache, allow rwx pool=vms, allow rwx 
>> pool=vms-cache, allow rwx pool=images, allow rwx pool=images-cache
>> client.radosgw.gateway
>> key: AQAU7uRccP06CBAA6zLFtDQoTstl8CNclYRugQ==
>> auid: 0
>> caps: [mon] allow rwx
>> caps: [osd] allow rwx
>> mgr.172.30.126.26
>> key: AQAr7uRclc52MhAA+GWCQEVnAHB01tMFpgJtTQ==
>> caps: [mds] allow *
>> caps: [mon] allow profile mgr
>> caps: [osd] allow *
>> mgr.172.30.126.27
>> key: AQAs7uRclkD2OBAAW/cUhcZEebZnQulqVodiXQ==
>> caps: [mds] allow *
>> caps: [mon] allow profile mgr
>> caps: [osd] allow *
>> mgr.172.30.126.28
>> key: AQAu7uRcT9OLBBAAZbEjb/N1NnZpIgfaAcThyQ==
>> caps: [mds] allow *
>> caps: [mon] allow profile mgr
>> caps: [osd] allow *
>>
>>
>> Can someone explain it to me?
>
>Your clients don't have the correct caps. See [1] or [2].
>
>
>> thanks!!
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>[1] 
>http://docs.ceph.com/docs/mimic/releases/luminous/#upgrade-from-jewel-or-kraken
>[2] 
>http://docs.ceph.com/docs/luminous/rbd/rados-rbd-cmds/#create-a-block-device-user
>
>-- 
>Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] rbd.ReadOnlyImage: [errno 30]

[解决]

Hi all,
We use ceph(luminous) + openstack(queens) in my test environment。The 
virtual machine does not start properly after the disaster test and the image 
of virtual machine can not create snap.The procedure is as follows：
#!/usr/bin/env python


import rados
import rbd
with rados.Rados(conffile='/etc/ceph/ceph.conf',rados_id='nova') as cluster:
with cluster.open_ioctx('vms') as ioctx:
rbd_inst = rbd.RBD()
print "start open rbd image"
with rbd.Image(ioctx, '10df4634-4401-45ca-9c57-f349b78da475_disk') as 
image:
print "start create snapshot"
image.create_snap('myimage_snap1')


when i run it ,it show readonlyimage,as follows:


start open rbd image
start create snapshot
Traceback (most recent call last):
  File "testpool.py", line 17, in 
image.create_snap('myimage_snap1')
  File "rbd.pyx", line 1790, in rbd.Image.create_snap 
(/builddir/build/BUILD/ceph-12.2.5/build/src/pybind/rbd/pyrex/rbd.c:15682)
rbd.ReadOnlyImage: [errno 30] error creating snapshot myimage_snap1 from 
10df4634-4401-45ca-9c57-f349b78da475_disk


but i run it with admin instead of nova,it is ok.


"ceph auth list"  as follow


installed auth entries:


osd.1
key: AQBL7uRcfuyxEBAAoK8JrQWMU6EEf/g83zKJjg==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.10
key: AQCV7uRcdsB9IBAAHbHHCaylVUZIPKFX20polQ==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.11
key: AQCW7uRcRIMRIhAAbXfLbQwijEO5ZQFWFZaO5w==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.2
key: AQBL7uRcfFMWDBAAo7kjQobGBbIHYfZkx45pOw==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.4
key: AQBk7uRc97CPOBAAK9IBJICvchZPc5p80bISsg==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.5
key: AQBk7uRcOdqaORAAkQeEtYsE6rLWLPhYuCTdHA==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.7
key: AQB97uRc+1eRJxAA34DImQIMFjzHSXZ25djp0Q==
caps: [mon] allow profile osd
caps: [osd] allow *
osd.8
key: AQB97uRcFilBJhAAXzSzNJsgwpobC8654Xo7Sw==
caps: [mon] allow profile osd
caps: [osd] allow *
client.admin
key: AQAU7uRcNia+BBAA09mOYdX+yJWbLCjcuMih0A==
auid: 0
caps: [mds] allow
caps: [mgr] allow *
caps: [mon] allow *
caps: [osd] allow *
client.cinder
key: AQBp7+RcOzPHGxAA7azgyayVu2RRNWJ7JxSJEg==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
pool=volumes, allow rwx pool=volumes-cache, allow rwx pool=vms, allow rwx 
pool=vms-cache, allow rx pool=images, allow rx pool=images-cache
client.cinder-backup
key: AQBq7+RcVOwGNRAAiwJ59ZvAUc0H4QkVeN82vA==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
pool=backups, allow rwx pool=backups-cache
client.glance
key: AQDf7uRc32hDBBAAkGucQEVTWqnIpNvihXf/Ng==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=images, 
allow rwx pool=images-cache
client.nova
key: AQDN7+RcqDABIxAAXnFcVjBp/S5GkgOy0wqB1Q==
caps: [mon] allow r
caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
pool=volumes, allow rwx pool=volumes-cache, allow rwx pool=vms, allow rwx 
pool=vms-cache, allow rwx pool=images, allow rwx pool=images-cache
client.radosgw.gateway
key: AQAU7uRccP06CBAA6zLFtDQoTstl8CNclYRugQ==
auid: 0
caps: [mon] allow rwx
caps: [osd] allow rwx
mgr.172.30.126.26
key: AQAr7uRclc52MhAA+GWCQEVnAHB01tMFpgJtTQ==
caps: [mds] allow *
caps: [mon] allow profile mgr
caps: [osd] allow *
mgr.172.30.126.27
key: AQAs7uRclkD2OBAAW/cUhcZEebZnQulqVodiXQ==
caps: [mds] allow *
caps: [mon] allow profile mgr
caps: [osd] allow *
mgr.172.30.126.28
key: AQAu7uRcT9OLBBAAZbEjb/N1NnZpIgfaAcThyQ==
caps: [mds] allow *
caps: [mon] allow profile mgr
caps: [osd] allow *
 


Can someone explain it to me? 
thanks!!



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] auth: could not find secret_id=6403

[解决]

Hi all,
 we use ceph(hammer) + openstack(mitaka) in my datacenter and there are 300 
osds and 3. Because the accident datacenter is powered off, all the servers are 
shut down. when power returns to normal ,we start 3 mon service at first, About 
two hours later we start  500 osd service,and later cluster is ok.
but one day later, there are "auth: could not find secret_id=6403" error on 
several osd hosts,and there are op is blocked 


2019-05-27 19:23:44.316416 7fb75451e700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.210:6816/1004678 pipe(0x128f5000 sd=586 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f4f8c0).accept connect_seq 5 vs existing 5 state standby
2019-05-27 19:23:44.316519 7fb755029700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.203:6834/1004655 pipe(0x173dd000 sd=584 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27472ec0).accept connect_seq 8 vs existing 7 state standby
2019-05-27 19:23:44.316561 7fb788d5c700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.211:6808/5206 pipe(0x144db000 sd=587 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f51700).accept connect_seq 7 vs existing 7 state standby
2019-05-27 19:23:44.316656 7fb77c198700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.219:6818/8946 pipe(0x1f575000 sd=588 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f4e580).accept connect_seq 7 vs existing 7 state standby
2019-05-27 19:23:44.316719 7fb78aa79700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.213:6810/5387 pipe(0x1f57 sd=93 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f512e0).accept connect_seq 5 vs existing 5 state standby
2019-05-27 19:23:44.316852 7fb754922700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.203:6828/1004835 pipe(0x128f sd=585 :6832 s=0 pgs=0 cs=0 l=0 
c=0x21d65b20).accept connect_seq 4 vs existing 3 state standby
2019-05-27 19:23:44.316929 7fb788d5c700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.211:6808/5206 pipe(0x144db000 sd=587 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f51700).accept connect_seq 8 vs existing 7 state standby
2019-05-27 19:23:44.317004 7fb75451e700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.210:6816/1004678 pipe(0x128f5000 sd=586 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f4f8c0).accept connect_seq 6 vs existing 5 state standby
2019-05-27 19:23:44.317148 7fb78aa79700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.213:6810/5387 pipe(0x1f57 sd=93 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f512e0).accept connect_seq 6 vs existing 5 state standby
2019-05-27 19:23:44.317207 7fb77c198700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.219:6818/8946 pipe(0x1f575000 sd=588 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f4e580).accept connect_seq 8 vs existing 7 state standby
2019-05-27 19:28:51.828430 7fb756e47700  0 auth: could not find secret_id=6403
2019-05-27 19:28:51.828446 7fb756e47700  0 cephx: verify_authorizer could not 
get service secret for service osd secret_id=6403
2019-05-27 19:28:51.828453 7fb756e47700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f50aa0).accept: got bad authorizer
2019-05-27 19:28:51.829282 7fb756e47700  0 auth: could not find secret_id=6403
2019-05-27 19:28:51.829296 7fb756e47700  0 cephx: verify_authorizer could not 
get service secret for service osd secret_id=6403
2019-05-27 19:28:51.829303 7fb756e47700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.216:6816/28337 pipe(0x21ba7000 sd=145 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f4f4a0).accept: got bad authorizer
2019-05-27 19:28:52.030139 7fb756e47700  0 auth: could not find secret_id=6403
2019-05-27 19:28:52.030153 7fb756e47700  0 cephx: verify_authorizer could not 
get service secret for service osd secret_id=6403
2019-05-27 19:28:52.030161 7fb756e47700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.216:6816/28337 pipe(0x20be9000 sd=145 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f51860).accept: got bad authorizer
2019-05-27 19:28:52.431002 7fb756e47700  0 auth: could not find secret_id=6403
2019-05-27 19:28:52.431017 7fb756e47700  0 cephx: verify_authorizer could not 
get service secret for service osd secret_id=6403
2019-05-27 19:28:52.431024 7fb756e47700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f4f600).accept: got bad authorizer
2019-05-27 19:28:53.231883 7fb756e47700  0 auth: could not find secret_id=6403
2019-05-27 19:28:53.231896 7fb756e47700  0 cephx: verify_authorizer could not 
get service secret for service osd secret_id=6403
2019-05-27 19:28:53.231903 7fb756e47700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.216:6816/28337 pipe(0x21ba7000 sd=145 :6832 s=0 pgs=0 cs=0 l=0 
c=0x27f4e9a0).accept: got bad authorizer
2019-05-27 19:28:54.832790 7fb756e47700  0 auth: could not find secret_id=6403
2019-05-27 19:28:54.832805 7fb756e47700  0 cephx: verify_authorizer could not 
get service secret for service osd secret_id=6403
2019-05-27 19:28:54.832812 7fb756e47700  0 -- 10.22.9.197:6832/1005404 >> 
10.22.9.216:6816/28337 pipe(0x27616000 sd=145 :6832 s=0 pgs=0 cs=0 l=0 
c=0x16f579c0).accept: got bad authorizer
2019-05-27 19:28:58.033720 7fb756e47700  0 auth: could not find secret_id=6403

[ceph-users] master osd crash during scrub pg or scrub pg manually

[解决]

Good evening everyone.
My ceph is cross-compiled and runs on armv7l 32-bit development board.The ceph 
version is 10.2.3,The compiler version is 6.3.0.
After I placed an object in the rados cluster, I scrubed the object manually. 
At this time, the main osd crashed.
Here is the osd log:


 ceph version  ()
 1: (()+0x7a7de8) [0x7fd1dde8]
 2: (__default_sa_restorer()+0) [0xb68db3c0]
 3: (()+0x24309c) [0x7f7b909c]
 4: (std::_Rb_tree_iterator 
std::_Rb_tree, 
hobject_t::BitwiseComparator, std::allocator >::_M_emplace_hint_unique, std::tuple<> 
>(std::_Rb_tree_const_iterator, 
std::piecewise_construct_t const&, std::tuple&&, 
std::tuple<>&&)+0x48) [0x7f87eed8]
 5: (ScrubMap::decode(ceph::buffer::list::iterator&, long long)+0x2b8) 
[0x7fa31498]
 6: (PG::sub_op_scrub_map(std::shared_ptr)+0x1e8) [0x7f862db8]
 7: (ReplicatedPG::do_sub_op(std::shared_ptr)+0x274) [0x7f8acb78]
 8: (ReplicatedPG::do_request(std::shared_ptr&, 
ThreadPool::TPHandle&)+0x518) [0x7f8d201c]
 9: (OSD::dequeue_op(boost::intrusive_ptr, std::shared_ptr, 
ThreadPool::TPHandle&)+0x3c4) [0x7f783e6c]
 10: (PGQueueable::RunVis::operator()(std::shared_ptr&)+0x68) 
[0x7f78412c]
 11: (OSD::ShardedOpWQ::_process(unsigned int, 
ceph::heartbeat_handle_d*)+0x5d4) [0x7f79c664]
 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x764) 
[0x7fe01da8]
 13: (()+0x88ea18) [0x7fe04a18]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.


 0> 2018-03-16 11:26:39.186442 95fe5a30  2 -- 172.16.10.31:6800/6528 >> 
172.16.10.35:6789/0 pipe(0x86236000 sd=23 :41154 s=2 pgs=174 cs=1 l=1 
c=0x8631b7c0).reader got KEEPALIVE_ACK
--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   1/ 5 kinetic
   1/ 5 fuse
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent 1
  max_new 1000
  log_file /var/log/ceph/ceph-osd.1.log




I also debugged with gdb.Here is the gdb debugging information:


[Thread 0x95636a30 (LWP 7835) exited]


Thread 50 "tp_osd_tp" received signal SIGSEGV, Segmentation fault.
0x7f85c09c in std::__cxx11::basic_string::_Alloc_hider::_Alloc_hider (__a=..., __dat=, this=)
at /usr/include/c++/6.3.0/bits/basic_string.h:110
110/usr/include/c++/6.3.0/bits/basic_string.h: No such file or directory.
(gdb) where
#0  0x7f85c09c in std::__cxx11::basic_string::_Alloc_hider::_Alloc_hider (__a=..., __dat=, this=)
at /usr/include/c++/6.3.0/bits/basic_string.h:110
#1  std::__cxx11::basic_string::basic_string (__str=..., this=) at 
/usr/include/c++/6.3.0/bits/basic_string.h:399
#2  object_t::object_t (this=) at 
/usr/src/debug/ceph-src/10.2.3-r0/git/src/include/object.h:32
#3  hobject_t::hobject_t (this=0x859a1850, rhs=...) at 
/usr/src/debug/ceph-src/10.2.3-r0/git/src/common/hobject.h:97
#4  0x7f921ed8 in std::pair::pair(std::tuple&, std::tuple<>&, 
std::_Index_tuple<0u>, std::_Index_tuple<>) (
__tuple2=..., __tuple1=..., this=0x859a1850) at 
/usr/include/c++/6.3.0/tuple:1586
#5  std::pair::pair(std::piecewise_construct_t, std::tuple, std::tuple<>) 
(__second=..., __first=...,
this=0x859a1850) at /usr/include/c++/6.3.0/tuple:1575
#6  __gnu_cxx::new_allocator >::construct 
>(std::pair*, std::piecewise_construct_t 
const&, std::tuple&&, std::tuple<>&&) (
this=, __p=0x859a1850) at 
/usr/include/c++/6.3.0/ext/new_allocator.h:120
#7  std::allocator_traits > >::construct >(std::allocator >&, std::pair*, 
std::piecewise_construct_t const&, std::tuple&&, 
std::tuple<>&&) (__a=..., __p=) at 
/usr/include/c++/6.3.0/bits/alloc_traits.h:455
#8  std::_Rb_tree, 
hobject_t::BitwiseComparator, std::allocator >::_M_construct_node, std::tuple<> 
>(std::_Rb_tree_node*,