[ceph-users] [ceph-osd ] osd can not boot
Hi :
A server was just rebooted and the osd cant boot .The log is the following.
-3> 2020-01-19 17:39:25.904673 7f5b8e5e9d80 -1
bluestore(/var/lib/ceph/osd/ceph-44) _verify_csum bad crc32c/0x1000
checksum at blob offset 0x0, got 0xd2acc81f, expected 0x62cf539d,
device location [0xaee7c~1000], logical extent 0x0~1000, object
#-1:1406dc50:::osdmap.7390:0#
-2> 2020-01-19 17:39:25.904758 7f5b8e5e9d80 -1 osd.44 0 failed to
load OSD map for epoch 7390, got 0 bytes
-1> 2020-01-19 17:39:25.904788 7f5b8e5e9d80 -1 osd.44 7396
load_pgs: have pgid 9.58 at epoch 7390, but missing map. Crashing.
0> 2020-01-19 17:39:25.909760 7f5b8e5e9d80 -1
/home/zhaowei/release/rpmbuild/BUILD/ceph/src/osd/OSD.cc: In function
'void OSD::load_pgs()' thread 7f5b8e5e9d80 time 2020-01-19
17:39:25.904801
/home/zhaowei/release/rpmbuild/BUILD/ceph/src/osd/OSD.cc: 4099: FAILED
assert(0 == "Missing map in load_pgs")
so I tried to use ceph-objectstore-tool to look if I can find some clue
dump meta info
["meta",{"oid":"osdmap.7387","key":"","snapid":0,"hash":171666792,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7388","key":"","snapid":0,"hash":171663928,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7389","key":"","snapid":0,"hash":171664328,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7390","key":"","snapid":0,"hash":171663400,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7391","key":"","snapid":0,"hash":171663864,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7392","key":"","snapid":0,"hash":171665032,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7393","key":"","snapid":0,"hash":171664984,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7394","key":"","snapid":0,"hash":171665384,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7395","key":"","snapid":0,"hash":171664568,"max":0,"pool":-1,"namespace":"","max":0}]
["meta",{"oid":"osdmap.7396","key":"","snapid":0,"hash":171664456,"max":0,"pool":-1,"namespace":"","max":0}]
trying to get osdmap.7396 , and it is ok. But since the epoch in
superblock is 7390 , it is failed. I don't know what can cause this
issue? If the osdmap.7396 have been writen to disk successfully, so
why the older map was failed. The ceph version is 12.2.4。
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Infiniband backend OSD communication
>From my understanding, the basic idea is that ceph exchange rdma information(qp,gid and so) through ip address on rdma device, and then communicate with each other throng rdma. But in my tests, there seemed to be some issues in that codes. On Fri, Jan 3, 2020 at 2:24 AM Nathan Stratton wrote: > > I am working on upgrading my current ethernet only ceph cluster to a combined > ethernet frontend and infiniband backend. From my research I understand that > I set: > > ms_cluster_type = async+rdma > ms_async_rdma_device_name = mlx4_0 > > What I don't understand is how does ceph know how to reach each OSD over > RDMA? Do I have to run IPoIB on top of infiniband and use that for OSD > addresses? > > Is there a way to use infiniband on backend without IPoIB and just use rdma > verbs? > > ><> > nathan stratton > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Can I add existing rgw users to a tenant
Hello: We want to use rgw tenant as a group. But Can I add existing rgw users to a new tenant ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] which tool to use for benchmarking rgw s3, yscb or cosbench
Hi: I found cosbench is a very convenient tool for benchmaring rgw. But when I read papers , I found YCSB tool, https://github.com/brianfrankcooper/YCSB/tree/master/s3 . It seems that this is used for test cloud service , and seems a right tool for our service . Has anyone tried this tool ?How is it compared to cosbench ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] how to debug slow requests
Hi ceph users:
I was doing write benchmark, and found some io will be blocked for a
very long time. The following log is one op , it seems to wait for
replica to finish. My ceph version is 12.2.4, and the pool is 3+2 EC .
Does anyone give me some adives about how I sould debug next ?
{
"ops": [
{
"description": "osd_op(client.17985.0:670679 39.18
39:1a63fc5c:::benchmark_data_SH-IDC1-10-5-37-174_2917453_object670678:head
[set-alloc-hint object_size 1048576 write_size 1048576,write
0~1048576] snapc 0=[] ondisk+write+known_if_redirected e1135)",
"initiated_at": "2019-07-20 23:13:18.725466",
"age": 329.248875,
"duration": 329.248901,
"type_data": {
"flag_point": "waiting for sub ops",
"client_info": {
"client": "client.17985",
"client_addr": "10.5.137.174:0/1544466091",
"tid": 670679
},
"events": [
{
"time": "2019-07-20 23:13:18.725466",
"event": "initiated"
},
{
"time": "2019-07-20 23:13:18.726585",
"event": "queued_for_pg"
},
{
"time": "2019-07-20 23:13:18.726606",
"event": "reached_pg"
},
{
"time": "2019-07-20 23:13:18.726752",
"event": "started"
},
{
"time": "2019-07-20 23:13:18.726842",
"event": "waiting for subops from 4"
},
{
"time": "2019-07-20 23:13:18.743134",
"event": "op_commit"
},
{
"time": "2019-07-20 23:13:18.743137",
"event": "op_applied"
}
]
}
},
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] krbd: Can I only just update krbd module without updating kernal?
Hi: Because of some reasons, I can update the kernal to higher version. So I wonder if I can only just update krbd kernal module ? Has anyone done this before? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
