Hi,
I'm running a Ceph cluster on version 0.80.1
(a38fe1169b6d2ac98b427334c12d7cf81f809b74).
Here's roughly what happened: cluster has been created, pg and pgp num
have been increased step by step to (now) 1024 to have a fitting number
for the amount of OSDs and size. Then (days later) the size of the rbd
pool was increased from 2 to 3 and now the pool has some degraded PGs:
-------------------------- ceph -s -------------------------------------
root@storage001:~# ceph -s
cluster 75b96824-9527-4c21-a5b4-01964736463a
health HEALTH_WARN 44 pgs degraded; 68 pgs stuck unclean; recovery
6529/491939 objects degraded (1.327%)
monmap e2: 3 mons at
{storage001=10.10.10.51:6789/0,storage002=10.10.10.52:6789/0,storage003=10.10.10.53:6789/0},
election epoch 76, quorum 0,1,2 storage001,storage002,storage003
osdmap e2336: 16 osds: 16 up, 16 in
pgmap v273813: 1152 pgs, 3 pools, 667 GB data, 166 kobjects
1898 GB used, 18301 GB / 20200 GB avail
6529/491939 objects degraded (1.327%)
1084 active+clean
44 active+degraded
24 active+remapped
client io 7583 B/s wr, 1 op/s
-------------------------- ceph -s -------------------------------------
(also, the pgmap version is steadily increasing still)
ceph health detail output: http://pastebin.com/vVw5vTPv
As you can see in this list, some PGs are stuck degraded with only 2
copies and others are stuck in a remapped state (have already made a 3rd
copy, but in the wrong spot). But for those that went to size=3 already,
they haven't placed a copy in storage003 (no osd num 10-15) and I don't
see why.
------------------------- OSD tree ------------------------------------
# id weight type name up/down reweight
-1 19.72 root default
-2 9.05 host storage001
1 1.81 osd.1 up 1
2 1.81 osd.2 up 1
3 1.81 osd.3 up 1
4 1.81 osd.4 up 1
0 1.81 osd.0 up 1
-3 9.05 host storage002
5 1.81 osd.5 up 1
6 1.81 osd.6 up 1
7 1.81 osd.7 up 1
8 1.81 osd.8 up 1
9 1.81 osd.9 up 1
-4 1.62 host storage003
11 0.27 osd.11 up 1
12 0.27 osd.12 up 1
13 0.27 osd.13 up 1
14 0.27 osd.14 up 1
15 0.27 osd.15 up 1
10 0.27 osd.10 up 1
------------------------- OSD tree ------------------------------------
Lastly, here's the query status of one of the degraded PGs:
"recovery_state": [
{ "name": "Started\/Primary\/Active",
"enter_time": "2014-06-10 14:19:33.083132",
"might_have_unfound": [],
"recovery_progress": { "backfill_targets": [],
"waiting_on_backfill": [],
"last_backfill_started": "0\/\/0\/\/-1",
"backfill_info": { "begin": "0\/\/0\/\/-1",
"end": "0\/\/0\/\/-1",
"objects": []},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": { "pull_from_peer": [],
"pushing": []}},
"scrub": { "scrubber.epoch_start": "0",
"scrubber.active": 0,
"scrubber.block_writes": 0,
"scrubber.finalizing": 0,
"scrubber.waiting_on": 0,
"scrubber.waiting_on_whom": []}},
{ "name": "Started",
"enter_time": "2014-06-10 14:19:32.076453"}],
"agent_state": {}}
Its been stuck in this state for 4 days, so its apparently not a matter
of time. Measures already taken that didn't help:
for var in {0..15}; do ceph osd scrub $var; done;
#ceph expects an integer after scrub, so * didn't work.
after that completed:
for var in {0..15}; do ceph osd repair $var; done;
after the repaid operation finished (couldn't find a single PG that
didn't report "0 fixed"), we tried restarting the OSDs one by one (which
is why the recovery enter_time is so recent).
Any help would be greatly appreciated.
Regards,
Marc
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
