Re: [ceph-users] How to repair active+clean+inconsistent?

2018-11-14 Thread Brad Hubbard
You could try a 'rados get' and then a 'rados put' on the object to start with. On Thu, Nov 15, 2018 at 4:07 AM K.C. Wong wrote: > > So, I’ve issued the deep-scrub command (and the repair command) > and nothing seems to happen. > Unrelated to this issue, I have to take down some OSD to prepare >

Re: [ceph-users] How to repair active+clean+inconsistent?

2018-11-14 Thread K.C. Wong
So, I’ve issued the deep-scrub command (and the repair command) and nothing seems to happen. Unrelated to this issue, I have to take down some OSD to prepare a host for RMA. One of them happens to be in the replication group for this PG. So, a scrub happened indirectly. I now have this from “ceph

Re: [ceph-users] How to repair active+clean+inconsistent?

2018-11-11 Thread K.C. Wong
Thanks, Ashley. Should I expect the deep-scrubbing to start immediately? [root@mgmt01 ~]# ceph pg deep-scrub 1.65 instructing pg 1.65 on osd.62 to deep-scrub [root@mgmt01 ~]# ceph pg ls deep_scrub pg_stat objects mip degrmispunf bytes log disklog state state_stamp v

Re: [ceph-users] How to repair active+clean+inconsistent?

2018-11-11 Thread Brad Hubbard
On Mon, Nov 12, 2018 at 4:21 PM Ashley Merrick wrote: > > Your need to run "ceph pg deep-scrub 1.65" first Right, thanks Ashley. That's what the "Note that you may have to do a deep scrub to populate the output." part of my answer meant but perhaps I needed to go further? The system has a

Re: [ceph-users] How to repair active+clean+inconsistent?

2018-11-11 Thread Ashley Merrick
Your need to run "ceph pg deep-scrub 1.65" first On Mon, Nov 12, 2018 at 2:20 PM K.C. Wong wrote: > Hi Brad, > > I got the following: > > [root@mgmt01 ~]# ceph health detail > HEALTH_ERR 1 pgs inconsistent; 1 scrub errors > pg 1.65 is active+clean+inconsistent, acting [62,67,47] > 1 scrub

Re: [ceph-users] How to repair active+clean+inconsistent?

2018-11-11 Thread K.C. Wong
Hi Brad, I got the following: [root@mgmt01 ~]# ceph health detail HEALTH_ERR 1 pgs inconsistent; 1 scrub errors pg 1.65 is active+clean+inconsistent, acting [62,67,47] 1 scrub errors [root@mgmt01 ~]# rados list-inconsistent-obj 1.65 No scrub information available for pg 1.65 error 2: (2) No such

Re: [ceph-users] How to repair active+clean+inconsistent?

2018-11-11 Thread Brad Hubbard
What does "rados list-inconsistent-obj " say? Note that you may have to do a deep scrub to populate the output. On Mon, Nov 12, 2018 at 5:10 AM K.C. Wong wrote: > > Hi folks, > > I would appreciate any pointer as to how I can resolve a > PG stuck in “active+clean+inconsistent” state. This has >

[ceph-users] How to repair active+clean+inconsistent?

2018-11-11 Thread K.C. Wong
Hi folks, I would appreciate any pointer as to how I can resolve a PG stuck in “active+clean+inconsistent” state. This has resulted in HEALTH_ERR status for the last 5 days with no end in sight. The state got triggered when one of the drives in the PG returned I/O error. I’ve since replaced the