I get that, even though I think it should be handled more gracefuly. But is it expected to also lead to consistency issues like this?
I think this is exactly what we're hitting right now http://tracker.ceph.com/issues/6101 <http://tracker.ceph.com/issues/6101> except I have no idea why it also happens on a freshly backfilled OSD... Jan > On 25 Sep 2015, at 19:02, Somnath Roy <[email protected]> wrote: > > Yes, known issue, make sure your system open file limit is pretty high.. > > From: ceph-users [mailto:[email protected] > <mailto:[email protected]>] On Behalf Of Jan Schermer > Sent: Friday, September 25, 2015 4:42 AM > To: [email protected] <mailto:[email protected]> > Subject: [ceph-users] OSD reaching file open limit - known issues? > > Hi, > we recently migrated some of our nodes to Ubuntu 12, which helped everything > quite a bit. > > But we hit a snag where the upstart initscript would not set the file open > ulimit correctly and some OSDs ran out of fds. > > Some roblems manifested since then on the node where this happened such as > scrub errors (which were corrected - don't ask me how, I was sleeping :-)) - > but not two of the OSDs on this node started failing with SIGABORT: > > 2015-09-25 10:45:17.860461 361ea17e700 -1 osd.12 pg_epoch: 1209679 pg[4.3d85( > v 1209679'13913518 (1209090'13910508,1209679'13913518] local-les=1209679 > n=235 ec=3 les/c 1209679/1209679 1209678/1209678/1209678) [12,36,59] r=0 > lpr=1209678 mlcod 1209656'13913517 active+clean > snaptrimq=[857c0~1,857f4~1,85a43~2]] trim_objectcould not find coid > 783dbd85/rbd_data.1a785181f15746a.00000000000238df/857c0//4 > 2015-09-25 10:45:17.862019 361ea17e700 -1 osd/ReplicatedPG.cc > <http://replicatedpg.cc/>: In function 'ReplicatedPG::RepGather* > ReplicatedPG::trim_object(const hobject_t&)' thread 361ea17e700 time > 2015-09-25 10:45:17.860501 > osd/ReplicatedPG.cc <http://replicatedpg.cc/>: 1510: FAILED assert(0) > > ceph version 0.67.11-82-ge5b6eea (e5b6eea91cc37434f78a987d2dd1d3edd4a23f3f) > 1: (ReplicatedPG::trim_object(hobject_t const&)+0x150) [0x6e8bd0] > 2: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim > const&)+0x2e7) [0x6ee0d7] > 3: (boost::statechart::detail::reaction_result > boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, > ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na>, > (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_react_impl<boost::mpl::list<boost::statechart::custom_reaction<ReplicatedPG::SnapTrim>, > boost::statechart::transition<ReplicatedPG::Reset, > ReplicatedPG::NotTrimming, > boost::statechart::detail::no_context<ReplicatedPG::Reset>, > &(boost::statechart::detail::no_context<ReplicatedPG::Reset>::no_function(ReplicatedPG::Reset > const&))>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, > boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, > ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> > >(boost::statechart::simple_state<ReplicatedPG::TrimmingObjects, > ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>&, > boost::statechart::event_base const&, void const*)+0x96) [0x740fa6] > 4: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, > ReplicatedPG::NotTrimming, std::allocator<void>, > boost::statechart::null_exception_translator>::process_queued_events()+0x137) > [0x71bdf7] > 5: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer, > ReplicatedPG::NotTrimming, std::allocator<void>, > boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base > const&)+0x26) [0x71cfe6] > 6: (ReplicatedPG::snap_trimmer()+0x4ed) [0x6b59ad] > 7: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x790c54] > 8: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68c) [0x9a69cc] > 9: (ThreadPool::WorkThread::entry()+0x10) [0x9a7c20] > 10: (()+0x7e9a) [0x36258121e9a] > 11: (clone()+0x6d) [0x3625669638d] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to > interpret this. > (full log available on request, file system will be thrashed shortly so tell > me if it's helpful to look for something in there) > > Could this all be caused by the OSD running out of file descriptors? Is it > supposed to handle a problem like this (meaning both the assert that happened > now and the file descriptor limit) gracefuly? Or is it a known issue that > this could happen? > The thing about upstart is it apparently keeps restarting the OSD, which > makes the problem even worse. > > Luckily we caught this in time and it only happened on one node, so we are > thrashing all the OSDs here. > > Looks like a problem that could hit anyone, and if it actually damages data > then it could be pretty bad and maybe worth looking into - tell me what more > is needed. > > Config: > XFS filesystem > Ubuntu 12 with 3.14.37 kernel > FIEMAP disabled > Ceph Dumpling (0.67.11-82-ge5b6eea) > > Thanks > > Jan > > > PLEASE NOTE: The information contained in this electronic mail message is > intended only for the use of the designated recipient(s) named above. If the > reader of this message is not the intended recipient, you are hereby notified > that you have received this message in error and that any review, > dissemination, distribution, or copying of this message is strictly > prohibited. If you have received this communication in error, please notify > the sender by telephone or e-mail (as shown above) immediately and destroy > any and all copies of this message in your possession (whether hard copies or > electronically stored copies).
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
