There should be entries in the cluster log about this damage, so find them, and possibly use the timestamps to refer to the MDS log for more detail.
Was the cluster originally created with infernalis or earlier? We have seen similar issues with a much older cluster that had actually lost dirfrag objects from a probable bug on older osds a long time ago -- the "upgrade" in jewel is that it actively records and reports this kind of thing. You can use `find` with the inode numbers from damage ls to identify which paths are effected. This is potentially repairable depending on exactly what's wrong. John On 24 Sep 2016 10:28 p.m., "Simion Marius Rad" <[email protected]> wrote: > Hi All, > > I run a CephFS production setup on Jewel 10.2.2 (updated from Infernalis > 9.2.1) and recently my cluster status changed and now it show that metadata > is damaged. I output of the "damage ls" command looks like this: > > [{"damage_type":"dir_frag","id":9547171,"ino":1099624711247,"frag":"*"},{" > damage_type":"dir_frag","id":34079746,"ino":1099597125430," > frag":"*"},{"damage_type":"dir_frag","id":52946895,"ino": > 1099604583518,"frag":"*"},{"damage_type":"dir_frag","id": > 148496486,"ino":1099624711225,"frag":"*"},{"damage_type":" > dir_frag","id":200490809,"ino":1099625521620,"frag":"*"},{" > damage_type":"dir_frag","id":205557997,"ino":1099606991149, > "frag":"*"},{"damage_type":"dir_frag","id":258125406,"ino" > :1099618690902,"frag":"*"},{"damage_type":"dir_frag","id": > 510860270,"ino":1099618587040,"frag":"*"},{"damage_type":" > dir_frag","id":549759027,"ino":1099628142420,"frag":"*"},{" > damage_type":"dir_frag","id":585330960,"ino":1099624189139, > "frag":"*"},{"damage_type":"dir_frag","id":820358338,"ino" > :1099624712699,"frag":"*"},{"damage_type":"dir_frag","id": > 942744318,"ino":1099628140501,"frag":"*"},{"damage_type":" > dir_frag","id":973031988,"ino":1099549383938,"frag":"*"},{" > damage_type":"dir_frag","id":1031321231,"ino":1099625361195,"frag":"*"},{" > damage_type":"dir_frag","id":1045361182,"ino":1099608544391,"frag":"*"},{" > damage_type":"dir_frag","id":1072944406,"ino":1099624188089,"frag":"*"},{" > damage_type":"dir_frag","id":1074378376,"ino":1099624711044,"frag":"*"},{" > damage_type":"dir_frag","id":1264544266,"ino":1099625355604,"frag":"*"},{" > damage_type":"dir_frag","id":1265110397,"ino":1099618372284,"frag":"*"},{" > damage_type":"dir_frag","id":1267294756,"ino":1099608269825,"frag":"*"},{" > damage_type":"dir_frag","id":1443589698,"ino":1099628140270,"frag":"*"},{" > damage_type":"dir_frag","id":1469860849,"ino":1099623182378,"frag":"*"},{" > damage_type":"dir_frag","id":1630484148,"ino":1099620745725,"frag":"*"},{" > damage_type":"dir_frag","id":1710015528,"ino":1099618586389,"frag":"*"},{" > damage_type":"dir_frag","id":1723591292,"ino":1099609007327,"frag":"*"},{" > damage_type":"dir_frag","id":1724584256,"ino":1099624712896,"frag":"*"},{" > damage_type":"dir_frag","id":1806338857,"ino":1099622921186,"frag":"*"},{" > damage_type":"dir_frag","id":1871119128,"ino":1099618123212,"frag":"*"},{" > damage_type":"dir_frag","id":1886372044,"ino":1099609002836,"frag":"*"},{" > damage_type":"dir_frag","id":1940442786,"ino":1099552476715,"frag":"*"},{" > damage_type":"dir_frag","id":1984022309,"ino":1099622715593,"frag":"*"},{" > damage_type":"dir_frag","id":2015183937,"ino":1099622864619,"frag":"*"},{" > damage_type":"dir_frag","id":2019732391,"ino":1099618117660,"frag":"*"},{" > damage_type":"dir_frag","id":2078183820,"ino":1099628250792,"frag":"*"},{" > damage_type":"dir_frag","id":2086391724,"ino":1099607263818,"frag":"*"},{" > damage_type":"dir_frag","id":2130427380,"ino":1099625363955,"frag":"*"},{" > damage_type":"dir_frag","id":2256866899,"ino":1099618114228,"frag":"*"},{" > damage_type":"dir_frag","id":2270858311,"ino":1099624713061,"frag":"*"},{" > damage_type":"dir_frag","id":2317877380,"ino":1099608894918,"frag":"*"},{" > damage_type":"dir_frag","id":2328690339,"ino":1099618586543,"frag":"*"},{" > damage_type":"dir_frag","id":2352777654,"ino":1099624030883,"frag":"*"},{" > damage_type":"dir_frag","id":2575139768,"ino":1099608266704,"frag":"*"},{" > damage_type":"dir_frag","id":2630991415,"ino":1099628252432,"frag":"*"},{" > damage_type":"dir_frag","id":2642152150,"ino":1099607934173,"frag":"*"},{" > damage_type":"dir_frag","id":2812946108,"ino":1099607186041,"frag":"*"},{" > damage_type":"dir_frag","id":2831890366,"ino":1099549384188,"frag":"*"},{" > damage_type":"dir_frag","id":2885680303,"ino":1099622808358,"frag":"*"},{" > damage_type":"dir_frag","id":3025465470,"ino":1099618587002,"frag":"*"},{" > damage_type":"dir_frag","id":3056319918,"ino":1099624029233,"frag":"*"},{" > damage_type":"dir_frag","id":3067128782,"ino":1099618586985,"frag":"*"},{" > damage_type":"dir_frag","id":3103804005,"ino":1099622919068,"frag":"*"},{" > damage_type":"dir_frag","id":3120240218,"ino":1099608666625,"frag":"*"},{" > damage_type":"dir_frag","id":3253821953,"ino":1099607263367,"frag":"*"},{" > damage_type":"dir_frag","id":3271496269,"ino":1099606721904,"frag":"*"},{" > damage_type":"dir_frag","id":3394616906,"ino":1099621759230,"frag":"*"},{" > damage_type":"dir_frag","id":3448326398,"ino":1099549383876,"frag":"*"},{" > damage_type":"dir_frag","id":3527854419,"ino":1099628250053,"frag":"*"},{" > damage_type":"dir_frag","id":3534906866,"ino":1099625726024,"frag":"*"},{" > damage_type":"dir_frag","id":3599935903,"ino":1099607961924,"frag":"*"},{" > damage_type":"dir_frag","id":3704836323,"ino":1099622920994,"frag":"*"},{" > damage_type":"dir_frag","id":3744599290,"ino":1099618372557,"frag":"*"},{" > damage_type":"dir_frag","id":3795019652,"ino":1099625520946,"frag":"*"},{" > damage_type":"dir_frag","id":3804382536,"ino":1099609001266,"frag":"*"},{" > damage_type":"dir_frag","id":3808069251,"ino":1099608786414,"frag":"*"},{" > damage_type":"dir_frag","id":3853440337,"ino":1099627782649,"frag":"*"},{" > damage_type":"dir_frag","id":3979835690,"ino":1099624029147,"frag":"*"},{" > damage_type":"dir_frag","id":4008790181,"ino":1099609002882,"frag":"*"},{" > damage_type":"dir_frag","id":4016298145,"ino":1099628250702,"frag":"*"},{" > damage_type":"dir_frag","id":4030657237,"ino":1099623682582,"frag":"*"},{" > damage_type":"dir_frag","id":4053237797,"ino":1099608787037,"frag":"*"}] > > I also restarted the MDS and rerun an "ls -R" command and the status > reappeared. > Is there any chance that I can fix this ? > > Thank you, > Simion Rad > > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
