Great, I just recovered the first placement group from this error. To be sure, I ran a deep-scrub and that comes back clean.
Thanks for all your help. Regards, Jeff On Thu, Mar 17, 2016 at 11:58 AM, Samuel Just <[email protected]> wrote: > Oh, it's getting a stat mismatch. I think what happened is that on > one of the earlier repairs it reset the stats to the wrong value (the > orphan was causing the primary to scan two objects twice, which > matches the stat mismatch I see here). A pg repair repair will clear > that up. > -Sam > > On Thu, Mar 17, 2016 at 9:22 AM, Jeffrey McDonald <[email protected]> > wrote: > > Thanks Sam..... > > > > Since I have prepared a script for this, I decided to go ahead with the > > checks.....(patience isn't one of my extended attributes....) > > > > I've got a file that searches the full erasure encoded spaces and does > your > > checklist below. I have operated only on one PG so far, the 70.459 one > > that we've been discussing. There was only the one file that I found > to > > be out of place--the one we already discussed/found and it has been > removed. > > > > The pg is still marked as inconsistent. I've scrubbed it a couple of > times > > now and what I've seen is: > > > > 2016-03-17 09:29:53.202818 7f2e816f8700 0 log_channel(cluster) log > [INF] : > > 70.459 deep-scrub starts > > 2016-03-17 09:36:38.436821 7f2e816f8700 -1 log_channel(cluster) log > [ERR] : > > 70.459s0 deep-scrub stat mismatch, got 22319/22321 objects, 0/0 clones, > > 22319/22321 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, > > 68440088914/68445454633 bytes,0/0 hit_set_archive bytes. > > 2016-03-17 09:36:38.436844 7f2e816f8700 -1 log_channel(cluster) log > [ERR] : > > 70.459 deep-scrub 1 errors > > 2016-03-17 09:44:23.592302 7f2e816f8700 0 log_channel(cluster) log > [INF] : > > 70.459 deep-scrub starts > > 2016-03-17 09:47:01.237846 7f2e816f8700 -1 log_channel(cluster) log > [ERR] : > > 70.459s0 deep-scrub stat mismatch, got 22319/22321 objects, 0/0 clones, > > 22319/22321 dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts, > > 68440088914/68445454633 bytes,0/0 hit_set_archive bytes. > > 2016-03-17 09:47:01.237880 7f2e816f8700 -1 log_channel(cluster) log > [ERR] : > > 70.459 deep-scrub 1 errors > > > > > > Should the scrub be sufficient to remove the inconsistent flag? I took > the > > osd offline during the repairs. I've looked at files in all of the > osds > > in the placement group and I'm not finding any more problem files. The > > vast majority of files do not have the user.cephos.lfn3 attribute. > There > > are 22321 objects that I seen and only about 230 have the > user.cephos.lfn3 > > file attribute. The files will have other attributes, just not > > user.cephos.lfn3. > > > > Regards, > > Jeff > > > > > > On Wed, Mar 16, 2016 at 3:53 PM, Samuel Just <[email protected]> wrote: > >> > >> Ok, like I said, most files with _long at the end are *not orphaned*. > >> The generation number also is *not* an indication of whether the file > >> is orphaned -- some of the orphaned files will have ffffffffffffffff > >> as the generation number and others won't. For each long filename > >> object in a pg you would have to: > >> 1) Pull the long name out of the attr > >> 2) Parse the hash out of the long name > >> 3) Turn that into a directory path > >> 4) Determine whether the file is at the right place in the path > >> 5) If not, remove it (or echo it to be checked) > >> > >> You probably want to wait for someone to get around to writing a > >> branch for ceph-objectstore-tool. Should happen in the next week or > >> two. > >> -Sam > >> > > > > -- > > > > Jeffrey McDonald, PhD > > Assistant Director for HPC Operations > > Minnesota Supercomputing Institute > > University of Minnesota Twin Cities > > 599 Walter Library email: [email protected] > > 117 Pleasant St SE phone: +1 612 625-6905 > > Minneapolis, MN 55455 fax: +1 612 624-8861 > > > > > -- Jeffrey McDonald, PhD Assistant Director for HPC Operations Minnesota Supercomputing Institute University of Minnesota Twin Cities 599 Walter Library email: [email protected] 117 Pleasant St SE phone: +1 612 625-6905 Minneapolis, MN 55455 fax: +1 612 624-8861
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
