Hi Chris, What is the inconsistency that scrub reports in the log? My guess is that the simplest way to resolve this is to remove whichever copy you decide is invalid, but it depends on what the inconstency it is trying/failing to repair is.
Thanks! sage On Fri, 6 Sep 2013, Chris Dunlop wrote: > G'day, > > I'm getting an OSD crash on 0.56.7-1~bpo70+1 whilst trying to repair an OSD: > > http://tracker.ceph.com/issues/6233 > > ---- > ceph version 0.56.7 (14f23ab86b0058a8651895b3dc972a29459f3a33) > 1: /usr/bin/ceph-osd() [0x8530a2] > 2: (()+0xf030) [0x7f541ca39030] > 3: (gsignal()+0x35) [0x7f541b132475] > 4: (abort()+0x180) [0x7f541b1356f0] > 5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f541b98789d] > 6: (()+0x63996) [0x7f541b985996] > 7: (()+0x639c3) [0x7f541b9859c3] > 8: (()+0x63bee) [0x7f541b985bee] > 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x8fa9a7] > 10: (object_info_t::decode(ceph::buffer::list::iterator&)+0x29) [0x95b579] > 11: (object_info_t::object_info_t(ceph::buffer::list&)+0x180) [0x695ec0] > 12: (PG::repair_object(hobject_t const&, ScrubMap::object*, int, int)+0xc7) > [0x7646b7] > 13: (PG::scrub_process_inconsistent()+0x9bd) [0x76534d] > 14: (PG::scrub_finish()+0x4f) [0x76587f] > 15: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x10d6) [0x76cb96] > 16: (PG::scrub(ThreadPool::TPHandle&)+0x138) [0x76d7e8] > 17: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0xf) [0x70515f] > 18: (ThreadPool::worker(ThreadPool::WorkThread*)+0x992) [0x8f0542] > 19: (ThreadPool::WorkThread::entry()+0x10) [0x8f14d0] > 20: (()+0x6b50) [0x7f541ca30b50] > 21: (clone()+0x6d) [0x7f541b1daa7d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > ---- > > This occurs as a result of: > > # ceph pg dump | grep inconsistent > 2.12 2723 0 0 0 11311299072 159189 159189 > active+clean+inconsistent 2013-09-06 09:35:47.512119 20117'690441 > 20120'7914185 [6,7] [6,7] 20021'675967 2013-09-03 15:58:12.459188 > 19384'665404 2013-08-28 12:42:07.490877 > # ceph pg repair 2.12 > > Looking at PG::repair_object per line 12 of the backtrace, I can see a > dout(10) which should tell me the problem object: > > ---- > src/osd/PG.cc: > void PG::repair_object(const hobject_t& soid, ScrubMap::object *po, int > bad_peer, int ok_peer) > { > dout(10) << "repair_object " << soid << " bad_peer osd." << bad_peer << " > ok_peer osd." << ok_peer << dendl; > ... > } > ---- > > The 'ceph pg dump' output above tells me the primary osd is '6', so I > can increase the logging level to 10 on osd.6 to get the debug output, > and repair again: > > # ceph osd tell 6 injectargs '--debug_osd 0/10' > # ceph pg repair 2.12 > > I get the same OSD crash, but this time it logs the dout from above, > which shows the problem object: > > -1> 2013-09-06 09:34:45.142224 7f0ae94bd700 10 osd.6 pg_epoch: 20117 > pg[2.12( v 20117'690441 (20117'689440,20117'690441] local-les=20115 n=2722 > ec=1 les/c 20115/20115 20108/20112/20112) [6,7] r=0 lpr=20112 mlcod > 20117'690440 active+scrubbing+deep+repair] repair_object > 56987a12/rb.0.17d9b.2ae8944a.000000001e11/head//2 bad_peer osd.7 ok_peer osd.6 > 0> 2013-09-06 09:34:45.206621 7f0ae94bd700 -1 *** Caught signal > (Aborted) ** > > So... > > Firstly, is anyone interested in further investigating the problem to > fix the crash behaviour? > > And, what's the best way to fix the pool? > > Cheers, > > Chris > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
