On Thu, Sep 15, 2016 at 10:30 PM, Jim Kilborn <j...@kilborns.com> wrote:
> I have a replicated cache pool and metadata pool which reside on ssd drives, 
> with a size of 2, backed by a erasure coded data pool
> The cephfs filesystem was in a healthy state. I pulled an SSD drive, to 
> perform an exercise in osd failure.
> The cluster recognized the ssd failure, and replicated back to a healthy 
> state, but I got a message saying the mds0 Metadata damage detected.
>    cluster 62ed97d6-adf4-12e4-8fd5-3d9701b22b86
>      health HEALTH_ERR
>             mds0: Metadata damage detected
>             mds0: Client master01.div18.swri.org failing to respond to cache 
> pressure
>      monmap e2: 3 mons at 
> {ceph01=,ceph02=,ceph03=}
>             election epoch 24, quorum 0,1,2 
> ceph01,darkjedi-ceph02,darkjedi-ceph03
>       fsmap e25: 1/1/1 up {0=-ceph04=up:active}, 1 up:standby
>      osdmap e1327: 20 osds: 20 up, 20 in
>             flags sortbitwise
>       pgmap v11630: 1536 pgs, 3 pools, 100896 MB data, 442 kobjects
>             201 GB used, 62915 GB / 63116 GB avail
>                 1536 active+clean
> In the mds logs of the active mds, I see the following:
> 7fad0c4b2700  0 -- >> 
> pipe(0x7fad25885400 sd=56 :33513 s=1 pgs=0 cs=0 l=1 c=0x7fad2585f980).fault
> 7fad14add700  0 mds.beacon.darkjedi-ceph04 handle_mds_beacon no longer laggy
> 7fad101d3700  0 mds.0.cache.dir(10000016c08) _fetched missing object for [dir 
> 10000016c08 /usr/ [2,head] auth v=0 cv=0/0 ap=1+0+0 state=1073741952 f() n() 
> hs=0+0,ss=0+0 | waiter=1 authpin=1 0x7fad25ced500]
> 7fad101d3700 -1 log_channel(cluster) log [ERR] : dir 10000016c08 object 
> missing on disk; some files may be lost
> 7fad0f9d2700  0 -- >> 
> pipe(0x7fad25a4e800 sd=42 :0 s=1 pgs=0 cs=0 l=1 c=0x7fad25bd5180).fault
> 7fad14add700 -1 log_channel(cluster) log [ERR] : unmatched fragstat size on 
> single dirfrag 10000016c08, inode has f(v0 m2016-09-14 14:00:36.654244 
> 13=1+12), dirfrag has f(v0 m2016-09-14 14:00:36.654244 1=0+1)
> 7fad14add700 -1 log_channel(cluster) log [ERR] : unmatched rstat rbytes on 
> single dirfrag 10000016c08, inode has n(v77 rc2016-09-14 14:00:36.654244 
> b1533163206 48173=43133+5040), dirfrag has n(v77 rc2016-09-14 14:00:36.654244 
> 1=0+1)
> 7fad101d3700 -1 log_channel(cluster) log [ERR] : unmatched rstat on 
> 10000016c08, inode has n(v78 rc2016-09-14 14:00:36.656244 2=0+2), dirfrags 
> have n(v0 rc2016-09-14 14:00:36.656244 3=0+3)
> I’m not sure why the metadata got damaged, since its being replicated, but I 
> want to fix the issue, and test again. However, I cant figure out the steps 
> to repair the metadata.

Losing an object like that is almost certainly a sign that you've hit
a bug -- probably an OSD bug if it was the OSDs being disrupted while
the MDS daemons continued to run.

The subsequent "unmatched fragstat" etc messages are probably a red
herring where the stats are only bad because the object is missing,
not because of some other issue (http://tracker.ceph.com/issues/17284)

> I saw something about running a damage ls, but I can’t seem to find a more 
> detailed repair document. Any pointers to get the metadata fixed? Seems both 
> my mds daemons are running correctly, but that error bothers me. Shouldn’t 
> happen I think.

You can get the detail on what's damaged with "ceph tell mds.<id>
damage ls" -- this spits out JSON that you may well want to parse with
a tiny python script.

> I tried the following command, but it doesn’t understand it….
> ceph --admin-daemon /var/run/ceph/ceph-mds. ceph03.asok damage ls
> I then rebooted all 4 ceph servers simultaneously (another stress test), and 
> the ceph cluster came back up healthy, and the mds damaged status has been 
> cleared!!  I  then replaced the ssd, put it back into service, and let the 
> backfill complete. The cluster was fully healthy. I pulled another ssd, and 
> repeated this process, yet I never got the damaged mds messages. Was this 
> just a random metadata damage due to yanking a drive out? Is there any 
> lingering affects of the metadata that I need to address?

The MDS damage table is an ephemeral structure, so when you reboot it
will forget about the damage.  I would expect that doing a "ls -R" on
your filsystem will cause the damage to be detected again as it
traverses the filesystem, although if it doesn't then that would be a
sign that the "missing object" was actually a bug failing to find it
one time, rather than a bug where the object has really been lost.


> -          Jim
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
ceph-users mailing list

Reply via email to