related ceph bug: http://tracker.ceph.com/issues/19606
** Description changed: All three monitors have been crashing (repeatedly) after host with few OSDs had been rebooted. Trace from monitor: ... - -2> 2017-04-13 12:34:39.204681 7fd2857aa700 5 -- op tracker -- seq: 22, time: 2017-04-13 12:34:39.204681, event: osdmap:prepare_update, op: osd_boot(osd.21 booted 0 features 576460752032874495 v2022) - -1> 2017-04-13 12:34:39.204693 7fd2857aa700 5 -- op tracker -- seq: 22, time: 2017-04-13 12:34:39.204692, event: osdmap:prepare_boot, op: osd_boot(osd.21 booted 0 features 576460752032874495 v2022) - 0> 2017-04-13 12:34:39.213266 7fd2857aa700 -1 mon/OSDMonitor.cc: In function 'bool OSDMonitor::prepare_boot(MonOpRequestRef)' thread 7fd2857aa700 time 2017-04-13 12:34:39.204709 + -2> 2017-04-13 12:34:39.204681 7fd2857aa700 5 -- op tracker -- seq: 22, time: 2017-04-13 12:34:39.204681, event: osdmap:prepare_update, op: osd_boot(osd.21 booted 0 features 576460752032874495 v2022) + -1> 2017-04-13 12:34:39.204693 7fd2857aa700 5 -- op tracker -- seq: 22, time: 2017-04-13 12:34:39.204692, event: osdmap:prepare_boot, op: osd_boot(osd.21 booted 0 features 576460752032874495 v2022) + 0> 2017-04-13 12:34:39.213266 7fd2857aa700 -1 mon/OSDMonitor.cc: In function 'bool OSDMonitor::prepare_boot(MonOpRequestRef)' thread 7fd2857aa700 time 2017-04-13 12:34:39.204709 mon/OSDMonitor.cc: 2105: FAILED assert(osdmap.get_uuid(from) == m->sb.osd_fsid) - ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f) - 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55f6c17cc260] - 2: (OSDMonitor::prepare_boot(std::shared_ptr<MonOpRequest>)+0x1bd2) [0x55f6c1477e82] - 3: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x28b) [0x55f6c14aaa1b] - 4: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xb4f) [0x55f6c145b84f] - 5: (PaxosService::C_RetryMessage::_finish(int)+0x58) [0x55f6c145ce38] - 6: (C_MonOp::finish(int)+0x82) [0x55f6c14250c2] - 7: (Context::complete(int)+0x9) [0x55f6c14241a9] - 8: (void finish_contexts<Context>(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x1fb) [0x55f6c142a8db] - 9: (Paxos::finish_round()+0x287) [0x55f6c14512b7] - 10: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xe19) [0x55f6c1452499] - 11: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x250) [0x55f6c1452cc0] - 12: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xa38) [0x55f6c141e5c8] - 13: (Monitor::_ms_dispatch(Message*)+0x554) [0x55f6c141edc4] - 14: (Monitor::ms_dispatch(Message*)+0x23) [0x55f6c1441e93] - 15: (DispatchQueue::entry()+0xf2b) [0x55f6c18c1fab] - 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x55f6c17b25ad] - 17: (()+0x76fa) [0x7fd28de0e6fa] - 18: (clone()+0x6d) [0x7fd28c0c8b5d] - NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. + ceph version 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f) + 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55f6c17cc260] + 2: (OSDMonitor::prepare_boot(std::shared_ptr<MonOpRequest>)+0x1bd2) [0x55f6c1477e82] + 3: (OSDMonitor::prepare_update(std::shared_ptr<MonOpRequest>)+0x28b) [0x55f6c14aaa1b] + 4: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0xb4f) [0x55f6c145b84f] + 5: (PaxosService::C_RetryMessage::_finish(int)+0x58) [0x55f6c145ce38] + 6: (C_MonOp::finish(int)+0x82) [0x55f6c14250c2] + 7: (Context::complete(int)+0x9) [0x55f6c14241a9] + 8: (void finish_contexts<Context>(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x1fb) [0x55f6c142a8db] + 9: (Paxos::finish_round()+0x287) [0x55f6c14512b7] + 10: (Paxos::handle_last(std::shared_ptr<MonOpRequest>)+0xe19) [0x55f6c1452499] + 11: (Paxos::dispatch(std::shared_ptr<MonOpRequest>)+0x250) [0x55f6c1452cc0] + 12: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0xa38) [0x55f6c141e5c8] + 13: (Monitor::_ms_dispatch(Message*)+0x554) [0x55f6c141edc4] + 14: (Monitor::ms_dispatch(Message*)+0x23) [0x55f6c1441e93] + 15: (DispatchQueue::entry()+0xf2b) [0x55f6c18c1fab] + 16: (DispatchQueue::DispatchThread::entry()+0xd) [0x55f6c17b25ad] + 17: (()+0x76fa) [0x7fd28de0e6fa] + 18: (clone()+0x6d) [0x7fd28c0c8b5d] + NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. + + + Affected version: 10.2.6-0ubuntu0.16.04.1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1682424 Title: All monitors crashes after OSD host reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1682424/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
