Folks,
we are seeing very slow MDSes on a CephFS that has nearly nothing to do. Slow
as in „200 metadata requests per second“ slow when putting a bit of load onto
it. We managed to get a wallclock of the MDS:
[root@node01 gdbpmp]# ./gdbpmp.py -i gdbpmp.data
Thread: 7 (mds-rank-fin) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% Finisher::finisher_thread_entry()
+ 94.12% MDSIOContextBase::complete
|+ 88.24% MDSRank::respawn
||+ 88.24% Context::complete
|| + 88.24% MDSDaemon::respawn
|| + 88.24% ceph::logging::Log::dump_recent()
|| + 52.94%
ceph::logging::Log::_flush(std::vector<ceph::logging::ConcreteEntry,
std::allocator<ceph::logging::ConcreteEntry> >&, bool)
|| |+ 23.53% __snprintf_chk
|| ||+ 23.53% __vsnprintf_internal
|| || + 17.65% __vfprintf_internal
|| || + 5.88% __strchrnul_evex
|| || + 5.88% _IO_default_xsputn
|| |+ 17.65% ceph::logging::Log::_flush_logbuf()
|| ||+ 17.65%
ceph::logging::Log::_log_safe_write(std::basic_string_view<char,
std::char_traits<char> >)
|| || + 17.65% safe_write
|| || + 17.65% write
|| |+ 5.88% __tz_convert
|| | + 5.88% __tzfile_compute
|| | + 5.88% __strcmp_evex
|| + 5.88% __memmove_avx512_unaligned_erms
|+ 5.88% std::lock_guard<ceph::fair_mutex>::lock_guard
| + 5.88% ceph::fair_mutex::lock
| + 5.88%
std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
ceph::fair_mutex::lock()::{lambda()#1})
| + 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 5.88% pthread_cond_wait@@GLIBC_2.3.2
| + 5.88% __futex_abstimed_wait_common
+ 5.88% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 5.88% pthread_cond_wait@@GLIBC_2.3.2
+ 5.88% __futex_abstimed_wait_common
Thread: 10 (mds-rank-progr) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% MDSRank::ProgressThread::entry
+ 100.00%
std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex>,
MDSRank::ProgressThread::entry()::<lambda()> >
+ 100.00%
std::_V2::condition_variable_any::wait<std::unique_lock<ceph::fair_mutex> >
+ 88.24% std::condition_variable::wait(std::unique_lock<std::mutex>&)
|+ 88.24% pthread_cond_wait@@GLIBC_2.3.2
| + 88.24% __futex_abstimed_wait_common
+ 11.76%
std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex>
>::~_Unlock
+ 11.76% std::unique_lock<ceph::fair_mutex>::lock
+ 11.76% ceph::fair_mutex::lock
+ 11.76%
std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
ceph::fair_mutex::lock()::{lambda()#1})
+ 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 11.76% pthread_cond_wait@@GLIBC_2.3.2
+ 11.76% __futex_abstimed_wait_common
Thread: 11 (mds-log-trim) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% MDLog::log_trim_upkeep
+ 100.00%
std::_V2::condition_variable_any::wait_for<std::unique_lock<ceph::fair_mutex>,
long, std::ratio<1l, 1000l> >
+ 100.00%
std::_V2::condition_variable_any::wait_until<std::unique_lock<ceph::fair_mutex>,
std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l,
1000000000l> > >
+ 82.35%
std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex>
>::~_Unlock
|+ 82.35% std::unique_lock<ceph::fair_mutex>::lock
| + 82.35% ceph::fair_mutex::lock
| + 82.35%
std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
ceph::fair_mutex::lock()::{lambda()#1})
| + 82.35% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 82.35% pthread_cond_wait@@GLIBC_2.3.2
| + 82.35% __futex_abstimed_wait_common
+ 17.65% std::condition_variable::wait_until<std::chrono::duration<long,
std::ratio<1l, 1000000000l> > >
+ 17.65%
std::condition_variable::__wait_until_impl<std::chrono::duration<long,
std::ratio<1l, 1000000000l> > >
+ 17.65% std::__condvar::wait_until
+ 17.65% pthread_cond_clockwait@GLIBC_2.30
+ 17.65% __futex_abstimed_wait_common
Thread: 12 (mds-cache-trim) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% MDCache::upkeep_main
+ 70.59% std::scoped_lock<ceph::fair_mutex>::scoped_lock
|+ 70.59% ceph::fair_mutex::lock
| + 70.59%
std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
ceph::fair_mutex::lock()::{lambda()#1})
| + 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 70.59% pthread_cond_wait@@GLIBC_2.3.2
| + 70.59% __futex_abstimed_wait_common
+ 23.53% std::condition_variable::wait_for<long, std::ratio<1l,
1000000000l> >
|+ 23.53% std::condition_variable::wait_until<std::chrono::duration<long,
std::ratio<1l, 1000000000l> > >
| + 23.53%
std::condition_variable::__wait_until_impl<std::chrono::duration<long,
std::ratio<1l, 1000000000l> > >
| + 23.53% std::__condvar::wait_until
| + 23.53% pthread_cond_clockwait@GLIBC_2.30
| + 23.53% __futex_abstimed_wait_common
+ 5.88% MDCache::check_memory_usage
+ 5.88% MemoryModel::sample
+ 5.88% MemoryModel::_sample
+ 5.88% std::getline<char, std::char_traits<char>, std::allocator<char> >
+ 5.88% std::basic_istream<char, std::char_traits<char> >&
std::getline<char, std::char_traits<char>, std::allocator<char>
>(std::basic_istream<char, std::char_traits<char> >&,
std::__cxx11::basic_string<char
, std::char_traits<char>, std::allocator<char> >&, char)
+ 5.88% std::basic_filebuf<char, std::char_traits<char> >::underflow()
+ 5.88% std::__basic_file<char>::xsgetn(char*, long)
+ 5.88% read
Thread: 15 (safe_timer) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% CommonSafeTimerThread<std::mutex>::entry()
+ 100.00% CommonSafeTimer<std::mutex>::timer_thread()
+ 88.24% pthread_cond_clockwait@GLIBC_2.30
|+ 88.24% __futex_abstimed_wait_common
+ 11.76% std::condition_variable::wait(std::unique_lock<std::mutex>&)
+ 11.76% pthread_cond_wait@@GLIBC_2.3.2
+ 11.76% __futex_abstimed_wait_common
Thread: 17 (safe_timer) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% CommonSafeTimerThread<ceph::fair_mutex>::entry()
+ 100.00% CommonSafeTimer<ceph::fair_mutex>::timer_thread()
+ 70.59%
std::_V2::condition_variable_any::_Unlock<std::unique_lock<ceph::fair_mutex>
>::~_Unlock()
|+ 70.59% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 70.59% pthread_cond_wait@@GLIBC_2.3.2
| + 70.59% __futex_abstimed_wait_common
+ 29.41% pthread_cond_clockwait@GLIBC_2.30
+ 29.41% __futex_abstimed_wait_common
Thread: 20 (ms_dispatch) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% DispatchQueue::DispatchThread::entry()
+ 100.00% DispatchQueue::entry()
+ 100.00% MDSDaemon::ms_dispatch2
+ 94.12% std::lock_guard<ceph::fair_mutex>::lock_guard
|+ 94.12% ceph::fair_mutex::lock
| + 94.12%
std::condition_variable::wait<ceph::fair_mutex::lock()::{lambda()#1}>(std::unique_lock<std::mutex>&,
ceph::fair_mutex::lock()::{lambda()#1})
| + 94.12% std::condition_variable::wait(std::unique_lock<std::mutex>&)
| + 94.12% pthread_cond_wait@@GLIBC_2.3.2
| + 94.12% __futex_abstimed_wait_common
+ 5.88% MDSRankDispatcher::ms_dispatch
+ 5.88% MDSRank::_dispatch
+ 5.88% MDSRank::_advance_queues
+ 5.88% MDSContext::complete
+ 5.88% Context::complete
+ 5.88% Server::handle_client_readdir
+ 5.88% CInode::encode_inodestat
+ 5.88%
encode<optmetadata_singleton<optmetadata_client_t<std::allocator>,
std::allocator>, std::allocator>
+ 5.88%
optmetadata_multiton<optmetadata_singleton<optmetadata_client_t<std::allocator>,
std::allocator>, std::allocator>::encode
+ 5.88% ceph::buffer::v15_2_0::list::append_hole(unsigned int)
+ 5.88%
ceph::buffer::v15_2_0::list::refill_append_space(unsigned int)
+ 5.88% mempool::pool_t::adjust_count(long, long)
Thread: 21 (io_context_pool) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00%
std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::i
o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&,
(auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >
>::_M_run()
+ 100.00%
std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::io_context_pool::start(sh
ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()()
+ 100.00%
std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::io_context_pool::start(s
hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul,
1ul>(std::_Index_tuple<0ul, 1ul>)
+ 100.00%
std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >,
ceph::async::io_context_pool::start(short)::{lambda()#1
}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&,
ceph::async::io_context_pool::start(sho
rt)::{lambda()#1}&&)
+ 100.00% std::__invoke_impl<void,
make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::io_context_pool::start(short):
:{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
make_named_thread<ceph::async::io_context_pool::start(short)::{la
mbda()#1}>(std::basic_string_view<char, std::char_traits<char>
>,
ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&,
(auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta
rt(short)::{lambda()#1}&&)
+ 100.00%
make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >,
ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd
a(auto:1&&,
(auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
const
+ 100.00%
std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
+ 100.00%
std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
+ 100.00% std::__invoke_impl<void,
ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
+ 100.00%
ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const
+ 100.00% boost::asio::io_context::run
+ 100.00%
boost::asio::detail::scheduler::run(boost::system::error_code&) [clone
.constprop.0] [clone .isra.0]
+ 100.00% boost::asio::detail::scheduler::do_run_one
+ 94.12% boost::asio::detail::conditionally_enabled_event::wait
|+ 94.12%
boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock>
| + 94.12% pthread_cond_wait@@GLIBC_2.3.2
| + 94.12% __futex_abstimed_wait_common
+ 5.88% boost::asio::detail::scheduler_operation::complete
+ 5.88%
boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
0ul> const, void>, std::allocator<voi
d>,
boost::asio::detail::scheduler_operation>::do_complete(void*,
boost::asio::detail::scheduler_operation*, boost::system::error_code const&,
unsigned long)
+ 5.88%
boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,
0ul> const, void>::operator()()
+ 5.88%
boost::asio::detail::executor_op<boost::asio::detail::binder0<MonClient::handle_config(MConfig*)::{lambda()#2}>,
std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete
(void*, boost::asio::detail::scheduler_operation*,
boost::system::error_code const&, unsigned long) [clone .lto_priv.0]
+ 5.88%
ceph::common::ConfigProxy::set_mon_vals(ceph::common::CephContext*,
std::map<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::__cxx11::basic_string<cha
r, std::char_traits<char>, std::allocator<char> >,
std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const, std::__c
xx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > > > const&, std::function<bool
(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>
> co
nst&, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > const&)>) [clone .isra.0]
+ 5.88%
md_config_t::set_mon_vals(ceph::common::CephContext*, ConfigValues&,
ConfigTracker const&, std::map<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, std::_
_cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::less<void>,
std::allocator<std::pair<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocato
r<char> > const, std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > > > > const&, std::function<bool
(std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > const&,
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
const&)>)
+ 5.88% md_config_t::update_legacy_vals(ConfigValues&)
+ 5.88% md_config_t::update_legacy_val(ConfigValues&,
Option const&, std::variant<long ConfigValues::*, unsigned long
ConfigValues::*, std::__cxx11::basic_string<char, std::char_traits<char>, s
td::allocator<char> > ConfigValues::*, double
ConfigValues::*, bool ConfigValues::*, entity_addr_t ConfigValues::*,
entity_addrvec_t ConfigValues::*, uuid_d ConfigValues::*>)
+ 5.88% md_config_t::_get_val(ConfigValues const&,
Option const&, boost::container::small_vector<std::pair<Option const*,
std::variant<std::monostate, std::__cxx11::basic_string<char, std::cha
r_traits<char>, std::allocator<char> >,
unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t,
std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long
, std::ratio<1l, 1000l> >, Option::size_t,
uuid_d> const*>, 4ul, void, void>*, std::basic_ostream<char,
std::char_traits<char> >*) const
+ 5.88%
md_config_t::_get_val_nometa[abi:cxx11](ConfigValues const&, Option const&)
const
Thread: 22 (io_context_pool) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00%
std::thread::_State_impl<std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::i
o_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&,
(auto:2&&)...)#1}, ceph::async::io_context_pool::start(short)::{lambda()#1}> >
>::_M_run()
+ 100.00%
std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::io_context_pool::start(sh
ort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}> >::operator()()
+ 100.00%
std::thread::_Invoker<std::tuple<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::io_context_pool::start(s
hort)::{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}> >::_M_invoke<0ul,
1ul>(std::_Index_tuple<0ul, 1ul>)
+ 100.00%
std::__invoke<make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >,
ceph::async::io_context_pool::start(short)::{lambda()#1
}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&,
ceph::async::io_context_pool::start(sho
rt)::{lambda()#1}&&)
+ 100.00% std::__invoke_impl<void,
make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >, ceph::async::io_context_pool::start(short):
:{lambda()#1}&&)::{lambda(auto:1&&, (auto:2&&)...)#1},
ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
make_named_thread<ceph::async::io_context_pool::start(short)::{la
mbda()#1}>(std::basic_string_view<char, std::char_traits<char>
>,
ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambda(auto:1&&,
(auto:2&&)...)#1}&&, ceph::async::io_context_pool::sta
rt(short)::{lambda()#1}&&)
+ 100.00%
make_named_thread<ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::basic_string_view<char,
std::char_traits<char> >,
ceph::async::io_context_pool::start(short)::{lambda()#1}&&)::{lambd
a(auto:1&&,
(auto:2&&)...)#1}::operator()<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
const
+ 100.00%
std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
+ 100.00%
std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
+ 100.00% std::__invoke_impl<void,
ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other,
ceph::async::io_context_pool::start(short)::{lambda()#1}&&)
+ 100.00%
ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const
+ 100.00% boost::asio::io_context::run
+ 100.00%
boost::asio::detail::scheduler::run(boost::system::error_code&) [clone
.constprop.0] [clone .isra.0]
+ 100.00% boost::asio::detail::scheduler::do_run_one
+ 100.00% boost::asio::detail::conditionally_enabled_event::wait
+ 100.00%
boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock>
+ 100.00% pthread_cond_wait@@GLIBC_2.3.2
+ 100.00% __futex_abstimed_wait_common
Thread: 26 (msgr-worker-2) - 17 samples
+ 100.00% clone
+ 100.00% start_thread
+ 100.00% execute_native_thread_routine
+ 100.00% std::_Function_handler<void (),
NetworkStack::add_thread(Worker*)::{lambda()#1}>::_M_invoke(std::_Any_data
const&)
+ 100.00% EventCenter::process_events(unsigned int,
std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)
+ 94.12% EpollDriver::event_wait(std::vector<FiredFileEvent,
std::allocator<FiredFileEvent> >&, timeval*)
|+ 94.12% epoll_wait
+ 5.88% AsyncConnection::process()
+ 5.88% ProtocolV2::run_continuation(Ct<ProtocolV2>&)
+ 5.88% ProtocolV2::handle_message()
+ 5.88% decode_message(ceph::common::CephContext*, int,
ceph_msg_header&, ceph_msg_footer&, ceph::buffer::v15_2_0::list&,
ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&,
boost::intrusive_ptr<Conne
ction>)
+ 5.88% MMgrMap::decode_payload()
+ 5.88%
MgrMap::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)
+ 5.88%
MgrMap::ModuleInfo::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&)
+ 5.88%
std::enable_if<denc_traits<std::set<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocat
or<char> > >,
std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > >,
void>::supported&&(!denc_traits<std::set<std::__cxx11::basic_string<char,
std::cha
r_traits<char>, std::allocator<char> >,
std::less<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char,
std::char_trait
s<char>, std::allocator<char> > > >,
void>::need_contiguous), void>::type
ceph::decode<std::set<std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::less<std::__cx
x11::basic_string<char, std::char_traits<char>,
std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > > >, denc_traits<std::set<s
td::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char> >, std::less<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__c
xx11::basic_string<char, std::char_traits<char>,
std::allocator<char> > > >, void> >(std::set<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx1
1::basic_string<char, std::char_traits<char>,
std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char,
std::char_traits<char>, std::allocator<char> > > >&, ceph::buffer::v15_2_0::
list::iterator_impl<true>&)
+ 5.88% ceph::buffer::v15_2_0::ptr::release()
Sidenote: while recording this, the MDS became so slow that the MDS cluster
declared it dead … As can be seen, the MDS is waiting in futex to no end, but
how can we find out what it is waiting for? FS has many small files but nearly
no activity ….
Thanks in advance
Martin
--
Martin Gerhard Loschwitz
Geschäftsführer / CEO, True West IT Services GmbH
Phone: +49 2433 5253130
Mobile: +49 176 61832178
Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany
Legal: HRB 21985, Amtsgericht Mönchengladbach
VAT: DE363893844
True West IT Services GmbH is compliant with the GDPR regulation on data
protection and privacy in the European Union and the European Economic Area.
You can request the information on how we collect and process your private data
according to the law by contacting the email sender.
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
