On Fri, Sep 21, 2018 at 5:04 PM Thomas White <[email protected]> wrote:
> Hi all, > > > > I have recently performed a few tasks, namely purging several buckets from > our RGWs and added additional hosts into Ceph causing some data movement > for a rebalance. As this is now almost completed, I kicked off some deep > scrubs and one PG is now returning the following information: > > > > 2018-09-21 23:17:59.717286 7f2f16796700 -1 log_channel(cluster) log [ERR] > : 14.1b18 shard 313 missing > 14:18daa344:::default.162489536.28__shadow_24TB%2f24TB%2fDESIGNTEAM%2fPROJECTS%2fDLA > Piper%2f_DLA_4033_ Global Thought Leadership%2fFilm%2f04 Assets%2fFootage > %ef%80%a2 Audio > Sync%2fDLA_Thought_Leadership_NYC%2fCam_1%2fA13I1483.MOV.2~5v3nJDNLONBYszy54ZXZZQgos1D4Ywp.359_6:head > > 2018-09-21 23:17:59.717292 7f2f16796700 -1 log_channel(cluster) log [ERR] > : 14.1b18 shard 665 missing > 14:18daa344:::default.162489536.28__shadow_24TB%2f24TB%2fDESIGNTEAM%2fPROJECTS%2fDLA > Piper%2f_DLA_4033_ Global Thought Leadership%2fFilm%2f04 Assets%2fFootage > %ef%80%a2 Audio > Sync%2fDLA_Thought_Leadership_NYC%2fCam_1%2fA13I1483.MOV.2~5v3nJDNLONBYszy54ZXZZQgos1D4Ywp.359_6:head > > 2018-09-21 23:17:59.885884 7f2f16796700 -1 log_channel(cluster) log [ERR] > : 14.1b18 shard 385 missing > 14:18daa344:::default.162489536.28__shadow_24TB%2f24TB%2fDESIGNTEAM%2fPROJECTS%2fDLA > Piper%2f_DLA_4033_ Global Thought Leadership%2fFilm%2f04 Assets%2fFootage > %ef%80%a2 Audio > Sync%2fDLA_Thought_Leadership_NYC%2fCam_1%2fA13I1483.MOV.2~5v3nJDNLONBYszy54ZXZZQgos1D4Ywp.359_6:head > > 2018-09-21 23:20:24.954402 7f2f16796700 -1 log_channel(cluster) log [ERR] > : 14.1b18 scrub stat mismatch, got 44026/44025 objects, 0/0 clones, > 44026/44025 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 > whiteouts, 45423386817/45419192513 bytes, 0/0 hit_set_archive bytes. > > 2018-09-21 23:20:24.954418 7f2f16796700 -1 log_channel(cluster) log [ERR] > : 14.1b18 scrub 1 missing, 0 inconsistent objects > > 2018-09-21 23:20:24.954421 7f2f16796700 -1 log_channel(cluster) log [ERR] > : 14.1b18 scrub 4 errors > > > > The object I recognise by name as belonging to a bucket purged earlier in > the day, and is meant to be deleted. > That's distressing. Has anything untoward happened to the cluster? Have you tried re-scrubbing the PG and seeing if it continues to believe that object is missing? > What would be the best means to resolve this inconsistency when the object > is supposed to be absent? > One easy way to is to write the object (with nonsense) and then delete it. -Greg > > > Kind Regards, > > > > Thomas > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
