Il 11/04/2014 12:40, Wido den Hollander ha scritto:
Have you tried to make the OSD as lost?
$ ceph osd lost 6
That will remove the PGs and the cluster should come back.
WOW it worked!
ok i found a new recipe for this case i want to share with you:
############################## STUCK+ACTIVE+CLEAN PGS ###
1) get the list of stucked pgs--> ceph health detail
2) forcing them to recreate -> ceph pg force_create_pg <PG id>
a) trick: --->>> for pg in `ceph health detail | grep stale | cut
-d' ' -f2`; do ceph pg force_create_pg $pg; done
3) stop OSD where the problem is:
a) to see launch--> ceph health detail
the value between [ ] is the osd
4) when the osd is down see the activity of the cluster (ceph -w) when
stabilized
launch: ceph osd lost 6 --yes-i-really-mean-it
5) see the activity of the cluster (ceph -w) when stabilized
again -> restart the osd
all should go fine..
my case:
[root@ds-07-01 mycephcluster]# for pg in `ceph health detail | grep
stale | cut -d' ' -f2`; do ceph pg force_create_pg $pg; done
Invalid command: pgid has no .
pg force_create_pg <pgid> : force creation of pg <pgid>
Error EINVAL: invalid command
pg 1.24 now creating, ok
pg 0.21 now creating, ok
pg 1.20 now creating, ok
pg 2.23 now creating, ok
pg 2.1f now creating, ok
pg 0.11 now creating, ok
pg 1.10 now creating, ok
pg 2.f now creating, ok
pg 0.1 now creating, ok
pg 1.0 now creating, ok
[root@ds-07-01 mycephcluster]# ceph osd lost 6
--yes-i-really-mean-itmarked osd lost in epoch 60
OTHER SHELL-->
[root@ds-07-04 ceph-6]# service ceph stop osd.6
=== osd.6 ===
Stopping Ceph osd.6 on ds-07-04...kill 5909...done
wait wait wait
[root@ds-07-04 ceph-6]# service ceph start osd.6
=== osd.6 ===
create-or-move updated item name 'osd.6' weight 7.16 at location
{host=ds-07-04,root=default} to crush map
Starting Ceph osd.6 on ds-07-04...
starting osd.6 at :/0 osd_data /var/lib/ceph/osd/ceph-6
/var/lib/ceph/osd/ceph-6/journal
[root@ds-07-04 ceph-6]#
result-->
BEFORE -----
root@ds-07-01 mycephcluster]# ceph -w
cluster 042a6983-b824-4c99-9ba3-03eebaf74afa
health HEALTH_WARN 25 pgs stale; 1 pgs stuck inactive; 16 pgs
stuck stale; 1 pgs stuck unclean; 1/8 in osds are down
monmap e3: 3 mons at
{ds-07-01=131.154.129.182:6789/0,ds-07-02=131.154.129.183:6789/0,ds-07-03=131.154.129.184:6789/0},
election epoch 22, quorum 0,1,2 ds-07-01,ds-07-02,ds-07-03
osdmap e57: 8 osds: 7 up, 8 in
pgmap v181: 192 pgs, 3 pools, 0 bytes data, 0 objects
290 MB used, 58641 GB / 58641 GB avail
1 creating
166 active+clean
25 stale+active+clean
DURING-----
[root@ds-07-01 mycephcluster]# ceph -w
cluster 042a6983-b824-4c99-9ba3-03eebaf74afa
health HEALTH_WARN 16 pgs stuck inactive; 16 pgs stuck unclean
monmap e3: 3 mons at
{ds-07-01=131.154.129.182:6789/0,ds-07-02=131.154.129.183:6789/0,ds-07-03=131.154.129.184:6789/0},
election epoch 22, quorum 0,1,2 ds-07-01,ds-07-02,ds-07-03
osdmap e59: 8 osds: 8 up, 8 in
pgmap v205: 192 pgs, 3 pools, 0 bytes data, 0 objects
285 MB used, 58641 GB / 58641 GB avail
16 creating <---- ALREADY FORCED TO RECREATE --->>
OSD IS STILL UP
176 active+clean
2014-04-11 12:51:29.933885 mon.0 [INF] pgmap v205: 192 pgs: 16 creating,
176 active+clean; 0 bytes data, 285 MB used, 58641 GB / 58641 GB avail
2014-04-11 12:52:01.055278 mon.0 [INF] osd.6 marked itself down
2014-04-11 12:52:01.198879 mon.0 [INF] osdmap e60: 8 osds: 7 up, 8 in
<<<<<----- OSD DOWN HERE
[CUT]
2014-04-11 12:52:26.850987 mon.0 [INF] pgmap v215: 192 pgs: 2 creating,
180 active+clean, 10 stale+active+clean; 0 bytes data, 284 MB used,
58641 GB / 58641 GB avail
2014-04-11 12:52:27.986149 mon.0 [INF] pgmap v216: 192 pgs: 2 creating,
180 active+clean, 10 stale+active+clean; 0 bytes data, 284 MB used,
58641 GB / 58641 GB avail <---- STABILIZED
2014-04-11 12:52:47.188774 mon.0 [INF] osd.6 131.154.129.185:6805/6312
boot <---- OSD UP HERE
2014-04-11 12:52:47.188913 mon.0 [INF] osdmap e63: 8 osds: 8 up, 8 in
2014-04-11 12:52:47.267271 mon.0 [INF] pgmap v217: 192 pgs: 2 creating,
180 active+clean, 10 stale+active+clean; 0 bytes data, 284 MB used,
58641 GB / 58641 GB avail
2014-04-11 12:52:48.358609 mon.0 [INF] osdmap e64: 8 osds: 8 up, 8 in
2014-04-11 12:52:48.609137 mon.0 [INF] pgmap v218: 192 pgs: 2 creating,
180 active+clean, 10 stale+active+clean; 0 bytes data, 284 MB used,
58641 GB / 58641 GB avail
[CUT]
2014-04-11 12:52:52.998858 mon.0 [INF] pgmap v221: 192 pgs: 2 creating,
180 active+clean, 10 stale+active+clean; 0 bytes data, 285 MB used,
58641 GB / 58641 GB avail
2014-04-11 12:52:54.041535 mon.0 [INF] pgmap v222: 192 pgs: 188
active+clean, 4 active+clean+replay; 0 bytes data, 285 MB used, 58641 GB
/ 58641 GB avail #### <---- YESS!!!
2014-04-11 12:53:38.128798 mon.0 [INF] pgmap v223: 192 pgs: 192
active+clean; 0 bytes data, 285 MB used, 58641 GB / 58641 GB avail <----
STABILIZED
AFTER-----
[root@ds-07-01 mycephcluster]# ceph -w
cluster 042a6983-b824-4c99-9ba3-03eebaf74afa
health HEALTH_OK
monmap e3: 3 mons at
{ds-07-01=131.154.129.182:6789/0,ds-07-02=131.154.129.183:6789/0,ds-07-03=131.154.129.184:6789/0},
election epoch 22, quorum 0,1,2 ds-07-01,ds-07-02,ds-07-03
osdmap e64: 8 osds: 8 up, 8 in
pgmap v226: 192 pgs, 3 pools, 0 bytes data, 0 objects
285 MB used, 58641 GB / 58641 GB avail
192 active+clean
I however advise you to run with at least rep size 2 for a pool.
yes but i'm testing the solution with the size pool = 1 because I have a
raid 6 disk attached trhought the san with 4Gb fiber channel link and
this device guarantee me the data availability and consistency
it is an old EMC storage environment and my boos has asked to see how it
works with ceph, before it was on production with GPFS.
Thanks a lot for helping!
there is some manual (the official manual isn't clear enought) to
better understand how to manage the "pools" and what meaning they have ?
At the moment I don't know how they are working on my cluster... for
example... i have 58641 GB avail of space.. and i have 3 pools how the
space is divided between the pools?
and, I have 192 pgs but i see that they are divided on the pools in 1/3
on data, 1/3 on metadata, 1/3 on rbd what is that meaning?
Some days ago i tried to test the speed of the entire system and i saw a
HUGE number of IOPS on the journal of each OSD that kills the
performance of the EMC storage... is there nothing that i can do (move
the journal on a partition not help i have already tried it)
best regards
Matteo
Thanks a lot
Matteo Favaro
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
