When I issue the "ceph pg repair 1.32" command I *do* see it reported in
the "ceph -w" output but I *do not* see any new messages about page 1.32 in
the log of osd.6 - even if I turn debug messages way up.
# ceph pg repair 1.32
instructing pg 1.32 on osd.6 to repair
(ceph -w shows)
2016-05-04 11:19:50.528355 mon.0 [INF] from='client.?
192.168.2.224:0/1341169978' entity='client.admin' cmd=[{"prefix": "pg
repair", "pgid": "1.32"}]: dispatch
---
Yes, I also noticed that there is only one copy of that pg. I have no idea
how it happened, but my pools (all of them) got set to replication size=1.
I re-set them back to the intended values as soon as I noticed it.
Currently the pools are configured like this:
# ceph osd pool ls detail
pool 0 'rbd' replicated size 2 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 64 pgp_num 64 last_change 349499 flags hashpspool
stripe_width 0
removed_snaps [1~d]
pool 1 'cephfs_data' replicated size 2 min_size 2 crush_ruleset 0
object_hash rjenkins pg_num 300 pgp_num 300 last_change 349490 lfor 25902
flags hashpspool crash_replay_interval 45 tiers 4 read_tier 4 write_tier 4
stripe_width 0
pool 2 'cephfs_metadata' replicated size 2 min_size 2 crush_ruleset 0
object_hash rjenkins pg_num 300 pgp_num 300 last_change 349503 flags
hashpspool stripe_width 0
pool 4 'ssd_cache' replicated size 2 min_size 1 crush_ruleset 0 object_hash
rjenkins pg_num 256 pgp_num 256 last_change 349490 flags
hashpspool,incomplete_clones tier_of 1 cache_mode writeback target_bytes
126701535232 target_objects 1000000 hit_set
bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 3600s x2
min_read_recency_for_promote 1 stripe_width 0
# ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-12 0.39999 root ssd_cache
-4 0.20000 host node11
8 0.20000 osd.8 up 1.00000 1.00000
-11 0.20000 host node13
0 0.20000 osd.0 up 1.00000 1.00000
-1 2.99997 root default
-7 0.29999 host node6
7 0.29999 osd.7 up 0.72400 1.00000
-8 0.23000 host node5
5 0.23000 osd.5 up 0.67996 1.00000
-6 0.45999 host node12
9 0.45999 osd.9 up 0.72157 1.00000
-10 0.67000 host node14
10 0.67000 osd.10 up 0.70659 1.00000
-13 0.67000 host node22
6 0.67000 osd.6 up 0.69070 1.00000
-15 0.67000 host node21
11 0.67000 osd.11 up 0.69788 1.00000
--
For the most part data in my ceph cluster is not critical. Also, I have a
recent backup. At this point I would be happy to resolve the pg problems
"any way possible" in order to get it working again. Can I just delete the
problematic pg (or the versions of it that are broken)?
I tried some commands to "accept the missing objects as lost" but it tells
me:
# ceph pg 1.32 mark_unfound_lost delete
pg has no unfound objects
The osd log for that is:
2016-05-04 11:31:03.742453 9b088350 0 osd.6 350327 do_command r=0
2016-05-04 11:31:03.763017 9b088350 0 osd.6 350327 do_command r=0 pg has
no unfound objects
2016-05-04 11:31:03.763066 9b088350 0 log_channel(cluster) log [INF] : pg
has no unfound objects
I also tried to "force create" the page:
# ceph pg force_create_pg 1.32
pg 1.32 now creating, ok
In that case, I do see a dispatch:
2016-05-04 11:32:42.073625 mon.4 [INF] from='client.?
192.168.2.224:0/208882728' entity='client.admin' cmd=[{"prefix": "pg
force_create_pg", "pgid": "1.32"}]: dispatch
2016-05-04 11:32:42.075024 mon.0 [INF] from='client.17514719 :/0'
entity='client.admin' cmd=[{"prefix": "pg force_create_pg", "pgid":
"1.32"}]: dispatch
2016-05-04 11:32:42.183389 mon.0 [INF] from='client.17514719 :/0'
entity='client.admin' cmd='[{"prefix": "pg force_create_pg", "pgid":
"1.32"}]': finished
That puts the page in a new state for a while:
# ceph health detail | grep 1.32
pg 1.32 is stuck inactive since forever, current state creating, last
acting []
pg 1.32 is stuck unclean since forever, current state creating, last acting
[]
But after a few minutes it returns to the previous state:
# ceph health detail | grep 1.32
pg 1.32 is stuck inactive for 160741.831891, current state
undersized+degraded+peered, last acting [6]
pg 1.32 is stuck unclean for 1093042.263678, current state
undersized+degraded+peered, last acting [6]
pg 1.32 is stuck undersized for 57229.481051, current state
undersized+degraded+peered, last acting [6]
pg 1.32 is stuck degraded for 57229.481382, current state
undersized+degraded+peered, last acting [6]
pg 1.32 is undersized+degraded+peered, acting [6]
Blade.
On Tue, May 3, 2016 at 10:45 AM, Oliver Dzombic <[email protected]>
wrote:
> Hi Blade,
>
> if you dont see anything in the logs, then you should raise the debug
> level/frequency.
>
> You must at least see, that the repair command has been issued ( started
> ).
>
> Also i am wondering about the [6] from your output.
>
> That means, that there is only 1 copy of it ( on osd.6 ).
>
> What is your setting for the minimal required copies ?
>
> osd_pool_default_min_size = ??
>
> And whats the setting for the to create copies ?
>
> osd_pool_default_size = ???
>
> Please give us the output of
>
> ceph osd pool ls detail
>
>
>
>
> --
> Mit freundlichen Gruessen / Best regards
>
> Oliver Dzombic
> IP-Interactive
>
> mailto:[email protected]
>
> Anschrift:
>
> IP Interactive UG ( haftungsbeschraenkt )
> Zum Sonnenberg 1-3
> 63571 Gelnhausen
>
> HRB 93402 beim Amtsgericht Hanau
> Geschäftsführung: Oliver Dzombic
>
> Steuer Nr.: 35 236 3622 1
> UST ID: DE274086107
>
>
> Am 03.05.2016 um 19:11 schrieb Blade Doyle:
> > Hi Oliver,
> >
> > Thanks for your reply.
> >
> > The problem could have been caused by crashing/flapping OSD's. The
> > cluster is stable now, but lots of pg problems remain.
> >
> > $ ceph health
> > HEALTH_ERR 4 pgs degraded; 158 pgs inconsistent; 4 pgs stuck degraded; 1
> > pgs stuck inactive; 10 pgs stuck unclean; 4 pgs stuck undersized; 4 pgs
> > undersized; recovery 1489/523934 objects degraded (0.284%); recovery
> > 2620/523934 objects misplaced (0.500%); 158 scrub errors
> >
> > Example: for pg 1.32 :
> >
> > $ ceph health detail | grep "pg 1.32"
> > pg 1.32 is stuck inactive for 13260.118985, current state
> > undersized+degraded+peered, last acting [6]
> > pg 1.32 is stuck unclean for 945560.550800, current state
> > undersized+degraded+peered, last acting [6]
> > pg 1.32 is stuck undersized for 12855.304944, current state
> > undersized+degraded+peered, last acting [6]
> > pg 1.32 is stuck degraded for 12855.305305, current state
> > undersized+degraded+peered, last acting [6]
> > pg 1.32 is undersized+degraded+peered, acting [6]
> >
> > I tried various things like:
> >
> > $ ceph pg repair 1.32
> > instructing pg 1.32 on osd.6 to repair
> >
> > $ ceph pg deep-scrub 1.32
> > instructing pg 1.32 on osd.6 to deep-scrub
> >
> > Its odd that I never do see any log on osd.6 about scrubbing or
> > repairing that pg (after waiting many hours). I attached "ceph pg
> > query" and a grep of osd logs for that page. If there is a better way
> > to provide large logs please let me know.
> >
> > For reference the last mention of that pg in the logs is:
> >
> > 2016-04-30 09:24:44.703785 975b9350 20 osd.6 349418 kicking pg 1.32
> > 2016-04-30 09:24:44.703880 975b9350 30 osd.6 pg_epoch: 349418 pg[1.32( v
> > 338815'7745 (20981'4727,338815'7745] local-les=349347 n=435 ec=17 les/c
> > 349347/349347 349418/349418/349418) [] r=-1 lpr=349418
> > pi=349346-349417/1 crt=338815'7743 lcod 0'0 inactive NOTIFY] lock
> >
> >
> > Suggestions appreciated,
> > Blade.
> >
> >
> >
> >
> > On Sat, Apr 30, 2016 at 9:31 AM, Blade Doyle <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > Hi Ceph-Users,
> >
> > Help with how to resolve these would be appreciated.
> >
> > 2016-04-30 09:25:58.399634 9b809350 0 log_channel(cluster) log
> > [INF] : 4.97 deep-scrub starts
> > 2016-04-30 09:26:00.041962 93009350 0 -- 192.168.2.52:6800/6640
> > <http://192.168.2.52:6800/6640> >> 192.168.2.32:0/3983425916
> > <http://192.168.2.32:0/3983425916> pipe(0x27406000 sd=111 :6800 s=0
> > pgs=0 cs=0 l=0 c=0x272da0a0).accept peer addr is really
> > 192.168.2.32:0/3983425916 <http://192.168.2.32:0/3983425916> (socket
> > is 192.168.2.32:38514/0 <http://192.168.2.32:38514/0>)
> > 2016-04-30 09:26:15.415883 9b809350 -1 log_channel(cluster) log
> > [ERR] : 4.97 deep-scrub stat mismatch, got 284/282 objects, 0/0
> > clones, 145/145 dirty, 0/0 omap, 4/2 hit_set_archive, 137/137
> > whiteouts, 365855441/365855441 bytes,340/340 hit_set_archive bytes.
> > 2016-04-30 09:26:15.415953 9b809350 -1 log_channel(cluster) log
> > [ERR] : 4.97 deep-scrub 1 errors
> > 2016-04-30 09:26:15.416425 9b809350 0 log_channel(cluster) log
> > [INF] : 4.97 scrub starts
> > 2016-04-30 09:26:15.682311 9b809350 -1 log_channel(cluster) log
> > [ERR] : 4.97 scrub stat mismatch, got 284/282 objects, 0/0 clones,
> > 145/145 dirty, 0/0 omap, 4/2 hit_set_archive, 137/137 whiteouts,
> > 365855441/365855441 bytes,340/340 hit_set_archive bytes.
> > 2016-04-30 09:26:15.682392 9b809350 -1 log_channel(cluster) log
> > [ERR] : 4.97 scrub 1 errors
> >
> > Thanks Much,
> > Blade.
> >
> >
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > [email protected]
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
