Re: [ceph-users] Ceph pg active+clean+inconsistent

According to the output you provided previously, OSD.51/90 might have unfound object. To shuffle object again, you could do: # ceph osd set noout; ceph osd set nodown # systemctl restart ceph-osd@51 * wait for OSD.51's process to be up # systemctl restart ceph-osd@90 * wait for OSD.90's

Re: [ceph-users] Ceph pg active+clean+inconsistent

Yes, it doesn't cause issues, but I don't see any way to "repair" the problem. One possible idea that I might do eventually if no solution is found is to copy the CephFS files in question and remove the ones with inconsistencies (which should remove the underlying rados objects). But it'd be

Re: [ceph-users] Ceph pg active+clean+inconsistent

# ceph pg debug unfound_objects_exist FALSE Andras On 01/03/2017 11:38 PM, Shinobu Kinjo wrote: Would you run: # ceph pg debug unfound_objects_exist On Wed, Jan 4, 2017 at 5:31 AM, Andras Pataki wrote: Here is the output of ceph pg query for one of hte

Re: [ceph-users] Ceph pg active+clean+inconsistent

Would you run: # ceph pg debug unfound_objects_exist On Wed, Jan 4, 2017 at 5:31 AM, Andras Pataki wrote: > Here is the output of ceph pg query for one of hte active+clean+inconsistent > PGs: > > { > "state": "active+clean+inconsistent", > "snap_trimq":

Re: [ceph-users] Ceph pg active+clean+inconsistent

The attributes for one of the inconsistent objects for the following scrub error: 2016-12-20 11:58:25.825830 7f3e1a4b1700 -1 log_channel(cluster) log [ERR] : deep-scrub 6.92c 6:34932257:::1000187bbb5.0009:head on disk size (0) does not match object info size (3014656) adjusted for ondisk

Re: [ceph-users] Ceph pg active+clean+inconsistent

Here is the output of ceph pg query for one of hte active+clean+inconsistent PGs: { "state": "active+clean+inconsistent", "snap_trimq": "[]", "epoch": 342982, "up": [ 319, 90, 51 ], "acting": [ 319, 90, 51 ],

Re: [ceph-users] Ceph pg active+clean+inconsistent

Plus do this as well: # rados list-inconsistent-obj ${PG ID} On Fri, Dec 23, 2016 at 7:08 PM, Brad Hubbard wrote: > Could you also try this? > > $ attr -l > ./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.0009__head_EED893F1__6 > > Take note of any of ceph._, ceph._@1,

Re: [ceph-users] Ceph pg active+clean+inconsistent

Could you also try this? $ attr -l ./DIR_1/DIR_F/DIR_3/DIR_9/DIR_8/1000187bb70.0009__head_EED893F1__6 Take note of any of ceph._, ceph._@1, ceph._@2, etc. For me on my test cluster it looks like this. $ attr -l

Re: [ceph-users] Ceph pg active+clean+inconsistent

Would you be able to execute ``ceph pg ${PG ID} query`` against that particular PG? On Wed, Dec 21, 2016 at 11:44 PM, Andras Pataki wrote: > Yes, size = 3, and I have checked that all three replicas are the same zero > length object on the disk. I think some

Re: [ceph-users] Ceph pg active+clean+inconsistent

Yes, size = 3, and I have checked that all three replicas are the same zero length object on the disk. I think some metadata info is mismatching what the OSD log refers to as "object info size". But I'm not sure what to do about it. pg repair does not fix it. In fact, the file this object

Re: [ceph-users] Ceph pg active+clean+inconsistent

Hi Andras, Iam not the experienced User but i guess you could have a look on this object on each related osd for the pg, compare them and delete the Different object. I assume you have size = 3. Then again pg repair. But be carefull iirc the replica will be recovered from the primary pg. Hth

Re: [ceph-users] Ceph pg active+clean+inconsistent

Hi cephers, Any ideas on how to proceed on the inconsistencies below? At the moment our ceph setup has 5 of these - in all cases it seems like some zero length objects that match across the three replicas, but do not match the object info size. I tried running pg repair on one of them, but

[ceph-users] Ceph pg active+clean+inconsistent

Hi everyone, Yesterday scrubbing turned up an inconsistency in one of our placement groups. We are running ceph 10.2.3, using CephFS and RBD for some VM images. [root@hyperv017 ~]# ceph -s cluster d7b33135-0940-4e48-8aa6-1d2026597c2f health HEALTH_ERR 1 pgs inconsistent