Okay, that's the right debugging but it wasn't quite as helpful on its own as I expected. Can you get a core dump (you might already have one, depending on system settings) of the crash and open it up with gdb and get a full backtrace? -Greg
On Mon, Oct 15, 2012 at 10:59 AM, Nick Couchman <[email protected]> wrote: > Well, hopefully this is still okay...8.5MB bzip2d, 230MB unzipped. > > -Nick > >>>> On 2012/10/15 at 11:47, Gregory Farnum <[email protected]> wrote: >> Yeah, zip it and post * somebody's going to have to download it and > do >> fun things. :) >> -Greg >> >> On Mon, Oct 15, 2012 at 10:43 AM, Nick Couchman > <[email protected]> >> wrote: >>> Anywhere in particular I should make it available? It's a little > over a >> million lines of debug in the file - I can put it on a pastebin, if > that >> works, or perhaps zip it up and throw it somewhere? >>> >>> -Nick >>> >>>>>> On 2012/10/15 at 11:26, Gregory Farnum <[email protected]> wrote: >>>> Something in the MDS log is bad or is poking at a bug in the code. > Can >>>> you turn on MDS debugging and restart a daemon and put that log >>>> somewhere accessible? >>>> debug mds = 20 >>>> debug journaler = 20 >>>> debug ms = 1 >>>> -Greg >>>> >>>> On Mon, Oct 15, 2012 at 10:02 AM, Nick Couchman > <[email protected]> >>>> wrote: >>>>> Well, both of my MDSs seem to be down right now, and then > continually >>>> segfault (every time I try to start them) with the following: >>>>> >>>>> ceph-mdsmon-a:~ # ceph-mds -n mds.b -c /etc/ceph/ceph.conf -f >>>>> starting mds.b at :/0 >>>>> *** Caught signal (Segmentation fault) ** >>>>> in thread 7fbe0d61d700 >>>>> ceph version 0.48.1argonaut >>>> (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c) >>>>> 1: ceph-mds() [0x7ef83a] >>>>> 2: (()+0xfd00) [0x7fbe15a0cd00] >>>>> 3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea] >>>>> 4: (MDLog::_replay_thread()+0x6b6) [0x6a2446] >>>>> 5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed] >>>>> 6: (()+0x7f05) [0x7fbe15a04f05] >>>>> 7: (clone()+0x6d) [0x7fbe14bc410d] >>>>> 2012-10-15 10:57:35.449161 7fbe0d61d700 -1 *** Caught signal > (Segmentation >>>> fault) ** >>>>> in thread 7fbe0d61d700 >>>>> >>>>> ceph version 0.48.1argonaut >>>> (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c) >>>>> 1: ceph-mds() [0x7ef83a] >>>>> 2: (()+0xfd00) [0x7fbe15a0cd00] >>>>> 3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea] >>>>> 4: (MDLog::_replay_thread()+0x6b6) [0x6a2446] >>>>> 5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed] >>>>> 6: (()+0x7f05) [0x7fbe15a04f05] >>>>> 7: (clone()+0x6d) [0x7fbe14bc410d] >>>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to >>>> interpret this. >>>>> >>>>> 0> 2012-10-15 10:57:35.449161 7fbe0d61d700 -1 *** Caught > signal >>>> (Segmentation fault) ** >>>>> in thread 7fbe0d61d700 >>>>> >>>>> ceph version 0.48.1argonaut >>>> (commit:a7ad701b9bd479f20429f19e6fea7373ca6bba7c) >>>>> 1: ceph-mds() [0x7ef83a] >>>>> 2: (()+0xfd00) [0x7fbe15a0cd00] >>>>> 3: (ESession::replay(MDS*)+0x3ea) [0x4dcfea] >>>>> 4: (MDLog::_replay_thread()+0x6b6) [0x6a2446] >>>>> 5: (MDLog::ReplayThread::entry()+0xd) [0x4cf5ed] >>>>> 6: (()+0x7f05) [0x7fbe15a04f05] >>>>> 7: (clone()+0x6d) [0x7fbe14bc410d] >>>>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to >>>> interpret this. >>>>> >>>>> Segmentation fault >>>>> >>>>> Anyone have any hints on recovering? I'm running 0.48.1argonaut - > I can >>>> attempt to upgrade to 0.48.2 and see if that helps, but I figured > if anyone >>>> can offer any insight as to what to do to get the replay to run > without >>>> segfaulting? >>>>> >>>>> >>>>> >>>>> -------- >>>>> This e-mail may contain confidential and privileged material for > the sole use >>>> of the intended recipient. If this email is not intended for you, > or you >> are >>>> not responsible for the delivery of this message to the intended > recipient, >>>> please note that this message may contain SEAKR Engineering > (SEAKR) >>>> Privileged/Proprietary Information. In such a case, you are > strictly >>>> prohibited from downloading, photocopying, distributing or > otherwise using >>>> this message, its contents or attachments in any way. If you have > received >>>> this message in error, please notify us immediately by replying to > this >> e-mail >>>> and delete the message from your mailbox. Information contained in > this >>>> message that does not relate to the business of SEAKR is neither > endorsed by >>>> nor attributable to SEAKR. >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe > ceph-devel" in >>>>> the body of a message to [email protected] >>>>> More majordomo info at > http://vger.kernel.org/majordomo-info.html >>> >>> >>> >>> -------- >>> >>> This e-mail may contain confidential and privileged material for the > sole use >> of the intended recipient. If this email is not intended for you, or > you are >> not responsible for the delivery of this message to the intended > recipient, >> please note that this message may contain SEAKR Engineering (SEAKR) >> Privileged/Proprietary Information. In such a case, you are strictly > >> prohibited from downloading, photocopying, distributing or otherwise > using >> this message, its contents or attachments in any way. If you have > received >> this message in error, please notify us immediately by replying to > this e-mail >> and delete the message from your mailbox. Information contained in > this >> message that does not relate to the business of SEAKR is neither > endorsed by >> nor attributable to SEAKR. >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" > in >> the body of a message to [email protected] >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -------- > This e-mail may contain confidential and privileged material for the sole use > of the intended recipient. If this email is not intended for you, or you are > not responsible for the delivery of this message to the intended recipient, > please note that this message may contain SEAKR Engineering (SEAKR) > Privileged/Proprietary Information. In such a case, you are strictly > prohibited from downloading, photocopying, distributing or otherwise using > this message, its contents or attachments in any way. If you have received > this message in error, please notify us immediately by replying to this > e-mail and delete the message from your mailbox. Information contained in > this message that does not relate to the business of SEAKR is neither > endorsed by nor attributable to SEAKR. -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
