I adjusted “osd max pg per osd hard ratio ” to 50.0 and left “mon max pg per
osd” at 5000 just to see if things would allow data movement. This worked, the
new pool I created finished its creation and spread out. I was able to then
copy the data from the existing pool into the new pool and delete the old one.
Used this process for copying the default pools:
ceph osd pool create .users.email.new 16
rados cppool .users.email .users.email.new
ceph osd pool delete .users.email .users.email --yes-i-really-really-mean-it
ceph osd pool rename .users.email.new .users.email
ceph osd pool application enable .users.email rgw
So at this point, I have recreated all the .rgw and .user pools except
.rgw.buckets with a pg_num of 16, which significantly reduced the pgs,
unfortunately, the incompletes are still there:
cluster:
health: HEALTH_WARN
Reduced data availability: 4 pgs inactive, 4 pgs incomplete
Degraded data redundancy: 4 pgs unclean
services:
mon: 3 daemons, quorum mon1,mon2,mon3
mgr: mon3(active), standbys: mon1, mon2
osd: 43 osds: 43 up, 43 in
data:
pools: 10 pools, 4240 pgs
objects: 8148k objects, 10486 GB
usage: 21536 GB used, 135 TB / 156 TB avail
pgs: 0.094% pgs not active
4236 active+clean
4 incomplete
The health page is showing blue instead of read on the donut chart, at one
point it jumped to green but its back to blue currently. There are no more ops
blocked/delayed either.
Thanks for assistance, it seems the cluster will play nice now. Any thoughts
on the stuck pgs? I ran a query on 11.720 and it shows:
"blocked_by": [
13,
27,
28
OSD 13 was acting strange so I wiped it and removed it from the cluster. This
was during the rebuild so I wasn’t aware of it blocking. Now I am trying to
figure out how a removed OSD is blocking. I went through the process to remove
it:
ceph osd crush remove
ceph auth del
ceph osd rm
I guess since the cluster was a hot mess at that point, its possible it was
borked and therefore the pg is borked. I am trying to avoid deleting the data
as there is data in the OSDs that are online.
-Brent
From: ceph-users [mailto:[email protected]] On Behalf Of Brent
Kennedy
Sent: Wednesday, January 10, 2018 12:20 PM
To: 'Janne Johansson' <[email protected]>
Cc: [email protected]
Subject: Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears
readonly )
I change “mon max pg per osd” to 5000 because when I changed it to zero, which
was supposed to disable it, it caused an issue where I couldn’t create any
pools. It would say 0 was larger than the minimum. I imagine that’s a bug, if
I wanted it disabled, then it shouldn’t use the calculation. I then set “osd
max pg per osd hard ratio ” to 5 after changing “mon max pg per osd” to 5000,
figuring 5*5000 would cover it. Perhaps not. I will adjust it to 30 and
restart the OSDs.
-Brent
From: Janne Johansson [mailto:[email protected]]
Sent: Wednesday, January 10, 2018 3:00 AM
To: Brent Kennedy <[email protected] <mailto:[email protected]> >
Cc: [email protected] <mailto:[email protected]>
Subject: Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears
readonly )
2018-01-10 8:51 GMT+01:00 Brent Kennedy <[email protected]
<mailto:[email protected]> >:
As per a previous thread, my pgs are set too high. I tried adjusting the “mon
max pg per osd” up higher and higher, which did clear the error(restarted
monitors and managers each time), but it seems that data simply wont move
around the cluster. If I stop the primary OSD of an incomplete pg, the cluster
just shows those affected pages as active+undersized+degraded:
I also adjusted “osd max pg per osd hard ratio ” to 5, but that didn’t seem to
trigger any data moved. I did restart the OSDs each time I changed it. The
data just wont finish moving. “ceph –w” shows this:
2018-01-10 07:49:27.715163 osd.20 [WRN] slow request 960.675164 seconds old,
received at 2018-01-10 07:33:27.039907: osd_op(client.3542508.0:4097 14.0
14.50e8d0b0 (undecoded) ondisk+write+known_if_redirected e125984) currently
queued_for_pg
Did you bump the ratio so that the PGs per OSD max * hard ratio actually became
more than the amount of PGs you had?
Last time you mailed the ratio was 25xx and the max was 200 which meant the
ratio would have needed to be far more than 5.0.
--
May the most significant bit of your life be positive.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
