Adjusting CRUSH weight shouldn't have caused this. Unfortunately the logs don't have a lot of hints — the thread that crashed doesn't have any output except for the Crashed state. If you can reproduce this with more debugging on we ought to be able to track it down; if not it seems we missed a strange upgrade issue that others haven't run into. :/ -Greg
On Tue, Aug 21, 2018 at 7:42 AM Kenneth Van Alstyne < [email protected]> wrote: > After looking into this further, is it possible that adjusting CRUSH > weight of the OSDs while running mis-matched versions of the ceph-osd > daemon across the cluster can cause this issue? Under certain > circumstances in our cluster, this may happen automatically on the > backend. I can’t duplicate the issue in a lab, but highly suspect this is > what happened. > > Thanks, > > -- > Kenneth Van Alstyne > Systems Architect > Knight Point Systems, LLC > Service-Disabled Veteran-Owned Business > 1775 Wiehle Avenue Suite 101 | Reston, VA 20190 > <https://maps.google.com/?q=1775+Wiehle+Avenue+Suite+101+%7C+Reston,+VA+20190&entry=gmail&source=g> > c: 228-547-8045 <(228)%20547-8045> f: 571-266-3106 <(571)%20266-3106> > www.knightpoint.com > DHS EAGLE II Prime Contractor: FC1 SDVOSB Track > GSA Schedule 70 SDVOSB: GS-35F-0646S > GSA MOBIS Schedule: GS-10F-0404Y > ISO 20000 / ISO 27001 / CMMI Level 3 > > Notice: This e-mail message, including any attachments, is for the sole > use of the intended recipient(s) and may contain confidential and > privileged information. Any unauthorized review, copy, use, disclosure, > or distribution is STRICTLY prohibited. If you are not the intended > recipient, please contact the sender by reply e-mail and destroy all copies > of the original message. > > On Aug 17, 2018, at 4:01 PM, Gregory Farnum <[email protected]> wrote: > > Do you have more logs that indicate what state machine event the crashing > OSDs received? This obviously shouldn't have happened, but it's a plausible > failure mode, especially if it's a relatively rare combination of events. > -Greg > > On Fri, Aug 17, 2018 at 4:49 PM Kenneth Van Alstyne < > [email protected]> wrote: > >> Hello all: >> I ran into an issue recently with one of my clusters when >> upgrading from 10.2.10 to 12.2.7. I have previously tested the upgrade in >> a lab and upgraded one of our five production clusters with no issues. On >> the second cluster, however, I ran into an issue where all OSDs that were >> NOT running Luminous yet (which was about 40% of the cluster at the time) >> all crashed with the same backtrace, which I have pasted below: >> >> === >> 0> 2018-08-13 17:35:13.160849 7f145c9ec700 -1 osd/PG.cc: In >> function >> 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, >> PG::RecoveryState::RecoveryMachine>::my_context)' thread 7f145c9ec700 time >> 2018-08-13 17:35:13.157319 >> osd/PG.cc: 5860: FAILED assert(0 == "we got a bad state machine event") >> >> ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x7f) [0x55b9bf08614f] >> 2: >> (PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, >> PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na, mpl_::na>, >> (boost::statechart::history_mode)0>::my_context)+0xc4) [0x55b9bea62db4] >> 3: (()+0x447366) [0x55b9bea9a366] >> 4: (boost::statechart::simple_state<PG::RecoveryState::Stray, >> PG::RecoveryState::Started, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, >> mpl_::na, mpl_::na, mpl_::na>, >> (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base >> const&, void const*)+0x2f7) [0x55b9beac8b77] >> 5: (boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, >> PG::RecoveryState::Initial, std::allocator<void>, >> boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base >> const&)+0x6b) [0x55b9beaab5bb] >> 6: (PG::handle_peering_event(std::shared_ptr<PG::CephPeeringEvt>, >> PG::RecoveryCtx*)+0x384) [0x55b9bea7db14] >> 7: (OSD::process_peering_events(std::__cxx11::list<PG*, >> std::allocator<PG*> > const&, ThreadPool::TPHandle&)+0x263) [0x55b9be9d1723] >> 8: (ThreadPool::BatchWorkQueue<PG>::_void_process(void*, >> ThreadPool::TPHandle&)+0x2a) [0x55b9bea1274a] >> 9: (ThreadPool::worker(ThreadPool::WorkThread*)+0xeb0) [0x55b9bf076d40] >> 10: (ThreadPool::WorkThread::entry()+0x10) [0x55b9bf077ef0] >> 11: (()+0x7507) [0x7f14e2c96507] >> 12: (clone()+0x3f) [0x7f14e0ca214f] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed >> to interpret this. >> === >> >> Once I restarted the impacted OSDs, which brought them up to 12.2.7, >> everything recovered just fine and the cluster is healthy. The only rub is >> that losing that many OSDs simultaneously caused a significant I/O >> disruption to the production servers for several minutes while I brought up >> the remaining OSDs. I have been trying to duplicate this issue in a lab >> again before continuing the upgrades on the other three clusters, but am >> coming up short. Has anyone seen anything like this and am I missing >> something obvious? >> >> Given how quickly the issue happened and the fact that I’m having a hard >> time reproducing this issue, I am limited in the amount of logging and >> debug information I have available, unfortunately. If it helps, all >> ceph-mon, ceph-mds, radosgw, and ceph-mgr daemons were running 12.2.7, >> while 30 of the 50 total ceph-osd daemons were also on 12.2.7 when the >> remaining 20 ceph-osd daemons (on 10.2.10) crashed. >> >> Thanks, >> >> -- >> Kenneth Van Alstyne >> Systems Architect >> Knight Point Systems, LLC >> Service-Disabled Veteran-Owned Business >> 1775 Wiehle Avenue Suite 101 | Reston, VA 20190 >> <https://maps.google.com/?q=1775+Wiehle+Avenue+Suite+101+%7C+Reston,+VA+20190&entry=gmail&source=g> >> c: 228-547-8045 <(228)%20547-8045> f: 571-266-3106 <(571)%20266-3106> >> www.knightpoint.com >> DHS EAGLE II Prime Contractor: FC1 SDVOSB Track >> GSA Schedule 70 SDVOSB: GS-35F-0646S >> GSA MOBIS Schedule: GS-10F-0404Y >> ISO 20000 / ISO 27001 / CMMI Level 3 >> >> Notice: This e-mail message, including any attachments, is for the sole >> use of the intended recipient(s) and may contain confidential and >> privileged information. Any unauthorized review, copy, use, disclosure, or >> distribution is STRICTLY prohibited. If you are not the intended recipient, >> please contact the sender by reply e-mail and destroy all copies of the >> original message. >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
