root@ceph-osd-1-2:~# rbd -p POOLNAME ls
rbd: pool rbd doesn't contain rbd images
root@ceph-osd-1-2:~#
2013/10/31 Иван Кудрявцев <[email protected]
<mailto:[email protected]>>
Hello, List.
I met very big trouble during ceph upgrade from bobtail to
cuttlefish.
My OSDs started to crash to stale so LA went to 100+ on node,
after I stop OSD I unable to launch it again because of errors.
So, I started to reformat OSDs and eventually meet that I have
"incomplete" PGs and It seems to me that I lost actual data (It
seems some drives I use is buggy and lost partition and filesystem
at all after reboot node as them was empty, and I placed them in
all my nodes in one time). I use ceph as source for RBD images, so
all my images are unaccessible and ceph health is:
root@ceph-osd-1-2:~# ceph -s
cluster e74c9e0b-40ce-45e4-904a-676cbaaf042d
health HEALTH_WARN 536 pgs backfill; 28 pgs backfilling; 546
pgs degraded; 19 pgs down; 76 pgs incomplete; 33 pgs peering; 23
pgs stale; 109 pgs stuck inactive; 23 pgs stuck stale; 673 pgs
stuck unclean; 11 requests are blocked > 32 sec; recovery
2769254/9737405 degraded (28.439%); recovering 76 o/s, 169MB/s;
mds cluster is degraded; clock skew detected on mon.0, mon.1
monmap e1: 3 mons at
{0=10.252.0.3:6789/0,1=10.252.0.4:6789/0,2=10.252.0.2:6789/0
<http://10.252.0.3:6789/0,1=10.252.0.4:6789/0,2=10.252.0.2:6789/0>},
election epoch 5486, quorum 0,1,2 0,1,2
osdmap e52800: 48 osds: 41 up, 41 in
pgmap v22294835: 1344 pgs: 671 active+clean, 18
active+remapped+wait_backfill, 150 active+degraded+wait_backfill,
12 down+peering, 14 stale+peering, 13 active+degraded+backfilling,
368 active+degraded+remapped+wait_backfill, 7 stale+down+peering,
74 incomplete, 15 active+degraded+remapped+backfilling, 2
stale+incomplete; 6883 GB data, 14627 GB used, 31364 GB / 46031 GB
avail; 2769254/9737405 degraded (28.439%); recovering 76 o/s,
169MB/s
mdsmap e564: 1/1/1 up {0=2=up:replay}, 2 up:standby
Actually, my images stored in RBD pool, but when I try to access
it I get:
root@ceph-osd-1-2:~# rbd ls
rbd: pool rbd doesn't contain rbd images
root@ceph-osd-1-2:~#
however, I can access images directly:
root@ceph-osd-1-2:~# rbd info lun-000000017
rbd image 'lun-000000017':
size 10000 MB in 320000 objects
order 15 (32768 bytes objects)
block_name_prefix: rb.0.3c07.238e1f29
format: 1
and what I really would like is to export data even with errors as
usual files and try to check and fix some using fsck. As I
understand I should someway fix "incomplete" pgs and "inactive"
pgs but I don't really understand how to do it?
root@ceph-osd-1-2:~# ceph pg dump | awk '{print $1, $14, $2
,$6/1024/1024 , $9}' | grep '\.' | grep inc
dumped all in format plain
2.1be [43,15,37] 1603 4018.79 incomplete
2.1aa [20,37,44] 4875 11930.5 incomplete
0.1a4 [45,37,27] 0 0 incomplete
2.1a1 [43,45,37] 988 2497.98 incomplete
2.1a0 [30,29,37] 0 0 incomplete
0.1a2 [30,29,37] 0 0 incomplete
1.1a3 [45,37,27] 0 0 incomplete
2.1a2 [45,37,27] 0 0 incomplete
1.1a1 [30,29,37] 0 0 incomplete
2.19d [18,42,24] 0 0 incomplete
0.19f [18,42,24] 0 0 incomplete
1.19e [18,42,24] 0 0 incomplete
0.198 [1,43,37] 0 0 incomplete
1.197 [1,43,37] 0 0 incomplete
2.196 [1,43,37] 0 0 incomplete
2.183 [28,30,37] 0 0 incomplete
2.15f [27,32,37] 0 0 incomplete
0.15c [42,31,37] 0 0 incomplete
1.15b [42,31,37] 0 0 incomplete
2.15a [42,31,37] 0 0 incomplete
2.137 [7,30,37] 0 0 incomplete
0.130 [41,31,25] 139 446.865 stale+incomplete
0.12d [31,37,27] 0 0 incomplete
1.12c [31,37,27] 0 0 incomplete
2.12b [31,37,27] 0 0 incomplete
0.124 [30,37,42] 0 0 incomplete
1.123 [30,37,42] 0 0 incomplete
2.122 [30,37,42] 0 0 incomplete
0.109 [31,37,27] 0 0 incomplete
1.108 [31,37,27] 0 0 incomplete
2.107 [31,37,27] 0 0 incomplete
2.f4 [12,37,20] 3132 7941.63 incomplete
2.ee <http://2.ee> [12,11,37] 3959 9825.61 incomplete
0.ec <http://0.ec> [32,37,42] 0 0 incomplete
1.eb [32,37,42] 0 0 incomplete
2.ea [32,37,42] 0 0 incomplete
2.de <http://2.de> [45,37,21] 1228 2960.71 incomplete
2.d8 [9,28,37] 832 2029.15 incomplete
2.c1 [13,30,37] 0 0 incomplete
2.b9 [32,2,37] 2971 7305.59 incomplete
2.a4 [30,22,33] 1343 3365.41 incomplete
2.9e [43,37,45] 651 1633.18 incomplete
2.9a [30,2,37] 0 0 incomplete
2.95 [6,43,37] 0 0 incomplete
1.96 [6,43,37] 0 0 incomplete
0.97 [6,43,37] 4 8.00209 incomplete
2.91 [47,28,1] 4518 11066.5 stale+incomplete
0.64 [32,37,8] 0 0 incomplete
1.63 [32,37,8] 0 0 incomplete
2.62 [32,37,8] 0 0 incomplete
0.59 [32,37,21] 311 1010.57 incomplete
2.55 [27,16,37] 0 0 incomplete
0.57 [27,16,37] 0 0 incomplete
1.56 [27,16,37] 0 0 incomplete
2.57 [32,37,21] 42 72.6953 incomplete
2.4c [11,37,30] 3315 8320.87 incomplete
0.49 [18,7,37] 0 0 incomplete
1.48 [18,7,37] 0 0 incomplete
2.47 [18,7,37] 0 0 incomplete
2.3d [28,44,37] 0 0 incomplete
0.3f [28,44,37] 0 0 incomplete
1.3e [28,44,37] 0 0 incomplete
2.2a [43,16,37] 5379 13236.4 incomplete
0.18 [32,42,37] 0 0 incomplete
1.17 [32,42,37] 0 0 incomplete
2.16 [32,42,37] 0 0 incomplete
2.d [45,29,37] 0 0 incomplete
1.e [45,29,37] 0 0 incomplete
0.f [45,29,37] 0 0 incomplete
2.c [18,20,37] 1208 3002.21 incomplete
2.9 [45,37,27] 0 0 incomplete
1.a [45,37,27] 0 0 incomplete
0.b [45,37,27] 28 80.5886 incomplete
2.7 [22,37,30] 2889 7272.09 incomplete
0.1 [31,37,27] 39 135.707 incomplete
1.0 [31,37,27] 0 0 incomplete
_______________________________________________
ceph-users mailing list
[email protected] <mailto:[email protected]>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757
