Hello,
I've run the profiler for about 5-6 minutes and this is what I've got:
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Using local file /usr/bin/ceph-mds.
Using local file /var/log/ceph/mds.kavehome-mgto-pro-fs01.profile.0009.heap.
Total: 400.0 MB
362.5 90.6% 90.6% 362.5 90.6%
ceph::buffer::create_aligned_in_mempool
20.4 5.1% 95.7% 29.8 7.5% CDir::_load_dentry
5.9 1.5% 97.2% 6.9 1.7% CDir::add_primary_dentry
4.7 1.2% 98.4% 4.7 1.2% ceph::logging::Log::create_entry
1.8 0.5% 98.8% 1.8 0.5% std::_Rb_tree::_M_emplace_hint_unique
1.8 0.5% 99.3% 2.2 0.5% compact_map_base::decode
0.6 0.1% 99.4% 0.7 0.2% CInode::add_client_cap
0.5 0.1% 99.5% 0.5 0.1% std::__cxx11::basic_string::_M_mutate
0.4 0.1% 99.6% 0.4 0.1% SimpleLock::more
0.4 0.1% 99.7% 0.4 0.1% MDCache::add_inode
0.3 0.1% 99.8% 0.3 0.1% CDir::add_to_bloom
0.2 0.1% 99.9% 0.2 0.1% CDir::steal_dentry
0.2 0.0% 99.9% 0.2 0.0% CInode::get_or_open_dirfrag
0.1 0.0% 99.9% 0.8 0.2% std::enable_if::type decode
0.1 0.0% 100.0% 0.1 0.0% ceph::buffer::list::crc32c
0.1 0.0% 100.0% 0.1 0.0% decode_message
0.0 0.0% 100.0% 0.0 0.0% OpTracker::create_request
0.0 0.0% 100.0% 0.0 0.0% TrackedOp::TrackedOp
0.0 0.0% 100.0% 0.0 0.0% std::vector::_M_emplace_back_aux
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_unique
0.0 0.0% 100.0% 0.0 0.0% CInode::add_dirfrag
0.0 0.0% 100.0% 0.0 0.0% MDLog::_prepare_new_segment
0.0 0.0% 100.0% 0.0 0.0% DispatchQueue::enqueue
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::push_back
0.0 0.0% 100.0% 0.0 0.0% Server::prepare_new_inode
0.0 0.0% 100.0% 365.6 91.4% EventCenter::process_events
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_copy
0.0 0.0% 100.0% 0.0 0.0% CDir::add_null_dentry
0.0 0.0% 100.0% 0.0 0.0% Locker::check_inode_max_size
0.0 0.0% 100.0% 0.0 0.0% CDentry::add_client_lease
0.0 0.0% 100.0% 0.0 0.0% CInode::project_inode
0.0 0.0% 100.0% 0.0 0.0% std::__cxx11::list::_M_insert
0.0 0.0% 100.0% 0.0 0.0% MDBalancer::handle_heartbeat
0.0 0.0% 100.0% 0.0 0.0% MDBalancer::send_heartbeat
0.0 0.0% 100.0% 0.0 0.0% C_GatherBase::C_GatherSub::complete
0.0 0.0% 100.0% 0.0 0.0% EventCenter::create_time_event
0.0 0.0% 100.0% 0.0 0.0% CDir::_omap_fetch
0.0 0.0% 100.0% 0.0 0.0% Locker::handle_inode_file_caps
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_equal
0.0 0.0% 100.0% 0.0 0.0% Locker::issue_caps
0.0 0.0% 100.0% 0.1 0.0% MDLog::_submit_thread
0.0 0.0% 100.0% 0.0 0.0% Journaler::_wait_for_flush
0.0 0.0% 100.0% 0.0 0.0% Journaler::wrap_finisher
0.0 0.0% 100.0% 0.0 0.0% MDSCacheObject::add_waiter
0.0 0.0% 100.0% 0.0 0.0% std::__cxx11::list::insert
0.0 0.0% 100.0% 0.0 0.0% std::__detail::_Map_base::operator[]
0.0 0.0% 100.0% 0.0 0.0% Locker::mark_updated_scatterlock
0.0 0.0% 100.0% 0.0 0.0% std::_Rb_tree::_M_insert_
0.0 0.0% 100.0% 0.0 0.0% alloc_ptr::operator->
0.0 0.0% 100.0% 0.0 0.0% ceph::buffer::list::append@5c1560
0.0 0.0% 100.0% 0.0 0.0%
ceph::buffer::malformed_input::~malformed_input
0.0 0.0% 100.0% 0.0 0.0% compact_set_base::insert
0.0 0.0% 100.0% 0.0 0.0% CDir::add_waiter
0.0 0.0% 100.0% 0.0 0.0% InoTable::apply_release_ids
0.0 0.0% 100.0% 0.0 0.0% InoTable::project_release_ids
0.0 0.0% 100.0% 2.2 0.5% InodeStoreBase::decode_bare
0.0 0.0% 100.0% 0.0 0.0% interval_set::erase
0.0 0.0% 100.0% 1.1 0.3% std::map::operator[]
0.0 0.0% 100.0% 0.0 0.0% Beacon::_send
0.0 0.0% 100.0% 0.0 0.0% MDSDaemon::reset_tick
0.0 0.0% 100.0% 0.0 0.0% MgrClient::send_report
0.0 0.0% 100.0% 0.0 0.0% Journaler::_do_flush
0.0 0.0% 100.0% 0.1 0.0% Locker::rdlock_start
0.0 0.0% 100.0% 0.0 0.0% MDCache::_get_waiter
0.0 0.0% 100.0% 0.0 0.0% CDentry::~CDentry
0.0 0.0% 100.0% 0.0 0.0% MonClient::schedule_tick
0.0 0.0% 100.0% 0.1 0.0% AsyncConnection::handle_write
0.0 0.0% 100.0% 0.1 0.0% AsyncConnection::prepare_send_message
0.0 0.0% 100.0% 365.5 91.4% AsyncConnection::process
0.0 0.0% 100.0% 0.3 0.1% AsyncConnection::send_message
0.0 0.0% 100.0% 0.0 0.0% AsyncConnection::tick
0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::_send_message
0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::send_message
0.0 0.0% 100.0% 0.0 0.0% AsyncMessenger::submit_message
0.0 0.0% 100.0% 0.0 0.0% Beacon::notify_health
0.0 0.0% 100.0% 0.2 0.1% CDentry::CDentry
0.0 0.0% 100.0% 0.0 0.0% CDentry::_mark_dirty
0.0 0.0% 100.0% 0.0 0.0% CDentry::auth_pin
0.0 0.0% 100.0% 0.0 0.0% CDentry::mark_dirty
0.0 0.0% 100.0% 0.0 0.0% CDentry::pop_projected_linkage
0.0 0.0% 100.0% 0.0 0.0% CDir::_mark_dirty
0.0 0.0% 100.0% 29.8 7.5% CDir::_omap_fetched
0.0 0.0% 100.0% 0.0 0.0% CDir::auth_pin
0.0 0.0% 100.0% 0.0 0.0% CDir::fetch
0.0 0.0% 100.0% 0.0 0.0% CDir::link_inode_work
0.0 0.0% 100.0% 0.0 0.0% CDir::link_primary_inode
0.0 0.0% 100.0% 0.0 0.0% CInode::_mark_dirty
0.0 0.0% 100.0% 0.0 0.0% CInode::add_waiter
0.0 0.0% 100.0% 0.0 0.0% CInode::auth_pin
0.0 0.0% 100.0% 0.4 0.1% CInode::choose_ideal_loner
0.0 0.0% 100.0% 0.4 0.1% CInode::encode_inodestat
0.0 0.0% 100.0% 0.0 0.0% CInode::mark_dirty
0.0 0.0% 100.0% 0.0 0.0% CInode::mark_dirty_parent
0.0 0.0% 100.0% 0.0 0.0% CInode::mark_dirty_rstat
0.0 0.0% 100.0% 0.0 0.0% CInode::pop_and_dirty_projected_inode
0.0 0.0% 100.0% 0.4 0.1% CInode::set_loner_cap
0.0 0.0% 100.0% 29.8 7.5% C_IO_Dir_OMAP_Fetched::finish
0.0 0.0% 100.0% 0.0 0.0% C_Locker_FileUpdate_finish::finish
0.0 0.0% 100.0% 0.0 0.0% C_MDL_CheckMaxSize::finish
0.0 0.0% 100.0% 0.0 0.0%
C_MDS_RetryRequest::~C_MDS_RetryRequest
0.0 0.0% 100.0% 0.0 0.0% C_MDS_inode_update_finish::finish
0.0 0.0% 100.0% 0.0 0.0% C_MDS_openc_finish::finish
0.0 0.0% 100.0% 0.0 0.0% C_MDS_session_finish::finish
0.0 0.0% 100.0% 0.0 0.0% C_OnFinisher::finish
0.0 0.0% 100.0% 1.2 0.3% Context::complete
0.0 0.0% 100.0% 3.7 0.9% DispatchQueue::DispatchThread::entry
0.0 0.0% 100.0% 3.7 0.9% DispatchQueue::entry
0.0 0.0% 100.0% 0.9 0.2% DispatchQueue::fast_dispatch
0.0 0.0% 100.0% 2.1 0.5% DispatchQueue::pre_dispatch
0.0 0.0% 100.0% 0.0 0.0% EMetaBlob::print
0.0 0.0% 100.0% 0.0 0.0% EventCenter::process_time_events
0.0 0.0% 100.0% 29.9 7.5% Finisher::finisher_thread_entry
0.0 0.0% 100.0% 0.4 0.1% FunctionContext::finish
0.0 0.0% 100.0% 0.0 0.0% Journaler::_flush
0.0 0.0% 100.0% 0.0 0.0% Journaler::_write_head
0.0 0.0% 100.0% 0.0 0.0% Journaler::flush
0.0 0.0% 100.0% 0.0 0.0% Journaler::wait_for_flush
0.0 0.0% 100.0% 0.0 0.0% Locker::_do_cap_release
0.0 0.0% 100.0% 0.0 0.0% Locker::_do_cap_update
0.0 0.0% 100.0% 0.0 0.0% Locker::_drop_non_rdlocks
0.0 0.0% 100.0% 0.0 0.0% Locker::_rdlock_kick
0.0 0.0% 100.0% 0.2 0.0% Locker::acquire_locks
0.0 0.0% 100.0% 0.0 0.0% Locker::adjust_cap_wanted
0.0 0.0% 100.0% 0.2 0.0% Locker::dispatch
0.0 0.0% 100.0% 0.0 0.0% Locker::drop_locks
0.0 0.0% 100.0% 0.4 0.1% Locker::eval
0.0 0.0% 100.0% 0.0 0.0% Locker::eval_gather
0.0 0.0% 100.0% 0.0 0.0% Locker::file_update_finish
0.0 0.0% 100.0% 0.0 0.0% Locker::handle_client_cap_release
0.0 0.0% 100.0% 0.2 0.0% Locker::handle_client_caps
0.0 0.0% 100.0% 0.0 0.0% Locker::handle_client_lease
0.0 0.0% 100.0% 0.0 0.0% Locker::handle_file_lock
0.0 0.0% 100.0% 0.0 0.0% Locker::handle_lock
0.0 0.0% 100.0% 0.0 0.0% Locker::issue_caps_set
0.0 0.0% 100.0% 0.0 0.0% Locker::issue_client_lease
0.0 0.0% 100.0% 0.6 0.2% Locker::issue_new_caps
0.0 0.0% 100.0% 0.0 0.0% Locker::local_wrlock_start
0.0 0.0% 100.0% 0.0 0.0% Locker::nudge_log
0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_eval
0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_mix
0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_nudge
0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_tick
0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_writebehind
0.0 0.0% 100.0% 0.0 0.0% Locker::scatter_writebehind_finish
0.0 0.0% 100.0% 0.0 0.0% Locker::send_lock_message@42d5b0
0.0 0.0% 100.0% 0.0 0.0% Locker::send_lock_message@42f2b0
0.0 0.0% 100.0% 0.0 0.0% Locker::share_inode_max_size
0.0 0.0% 100.0% 0.0 0.0% Locker::simple_lock
0.0 0.0% 100.0% 0.0 0.0% Locker::simple_sync
0.0 0.0% 100.0% 0.0 0.0% Locker::tick
0.0 0.0% 100.0% 0.0 0.0% Locker::try_eval@43da60
0.0 0.0% 100.0% 0.0 0.0% Locker::try_eval@441fb0
0.0 0.0% 100.0% 0.0 0.0% Locker::wrlock_finish
0.0 0.0% 100.0% 0.0 0.0% Locker::wrlock_force
0.0 0.0% 100.0% 0.0 0.0% Locker::wrlock_start
0.0 0.0% 100.0% 0.0 0.0% Locker::xlock_start
0.0 0.0% 100.0% 0.0 0.0% MClientCaps::print
0.0 0.0% 100.0% 0.0 0.0% MClientRequest::decode_payload
0.0 0.0% 100.0% 0.0 0.0% MClientRequest::print
0.0 0.0% 100.0% 0.0 0.0% MDBalancer::prep_rebalance
0.0 0.0% 100.0% 0.0 0.0% MDBalancer::proc_message
0.0 0.0% 100.0% 0.0 0.0% MDCache::check_memory_usage
0.0 0.0% 100.0% 0.2 0.1% MDCache::path_traverse
0.0 0.0% 100.0% 0.0 0.0% MDCache::predirty_journal_parents
0.0 0.0% 100.0% 0.0 0.0% MDCache::request_cleanup
0.0 0.0% 100.0% 0.0 0.0% MDCache::request_finish
0.0 0.0% 100.0% 0.0 0.0% MDCache::request_start
0.0 0.0% 100.0% 0.3 0.1% MDCache::trim
0.0 0.0% 100.0% 0.3 0.1% MDCache::trim_dentry
0.0 0.0% 100.0% 0.3 0.1% MDCache::trim_lru
0.0 0.0% 100.0% 0.0 0.0% MDCache::truncate_inode
0.0 0.0% 100.0% 0.1 0.0% MDLog::SubmitThread::entry
0.0 0.0% 100.0% 0.0 0.0% MDLog::_start_new_segment
0.0 0.0% 100.0% 0.0 0.0% MDLog::_submit_entry
0.0 0.0% 100.0% 0.0 0.0% MDLog::submit_entry
0.0 0.0% 100.0% 0.0 0.0% MDSCacheObject::finish_waiting
0.0 0.0% 100.0% 0.2 0.1% MDSCacheObject::get
0.0 0.0% 100.0% 1.7 0.4% MDSDaemon::ms_dispatch
0.0 0.0% 100.0% 0.0 0.0% MDSDaemon::tick
0.0 0.0% 100.0% 29.9 7.5% MDSIOContextBase::complete
0.0 0.0% 100.0% 0.7 0.2% MDSInternalContextBase::complete
0.0 0.0% 100.0% 0.0 0.0% MDSLogContextBase::complete
0.0 0.0% 100.0% 0.3 0.1% MDSRank::ProgressThread::entry
0.0 0.0% 100.0% 0.7 0.2% MDSRank::_advance_queues
0.0 0.0% 100.0% 1.7 0.4% MDSRank::_dispatch
0.0 0.0% 100.0% 1.3 0.3% MDSRank::handle_deferrable_message
0.0 0.0% 100.0% 0.0 0.0% MDSRank::send_message_client
0.0 0.0% 100.0% 0.0 0.0%
MDSRank::send_message_client_counted@2a9260
0.0 0.0% 100.0% 0.0 0.0%
MDSRank::send_message_client_counted@2a94f0
0.0 0.0% 100.0% 0.0 0.0%
MDSRank::send_message_client_counted@2b1920
0.0 0.0% 100.0% 0.0 0.0% MDSRank::send_message_mds
0.0 0.0% 100.0% 1.7 0.4% MDSRankDispatcher::ms_dispatch
0.0 0.0% 100.0% 0.4 0.1% MDSRankDispatcher::tick
0.0 0.0% 100.0% 0.0 0.0% MOSDOp::print
0.0 0.0% 100.0% 0.1 0.0% Message::encode
0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_rotating
0.0 0.0% 100.0% 0.0 0.0% MonClient::_check_auth_tickets
0.0 0.0% 100.0% 0.0 0.0% MonClient::_send_mon_message
0.0 0.0% 100.0% 0.0 0.0% MonClient::tick
0.0 0.0% 100.0% 0.0 0.0% MutationImpl::MutationImpl
0.0 0.0% 100.0% 0.1 0.0% MutationImpl::auth_pin
0.0 0.0% 100.0% 0.1 0.0% MutationImpl::pin
0.0 0.0% 100.0% 0.0 0.0% MutationImpl::start_locking
0.0 0.0% 100.0% 365.6 91.4% NetworkStack::get_worker
0.0 0.0% 100.0% 0.8 0.2%
ObjectOperation::C_ObjectOperation_decodevals::finish
0.0 0.0% 100.0% 0.1 0.0% Objecter::_op_submit
0.0 0.0% 100.0% 0.1 0.0% Objecter::_op_submit_with_budget
0.0 0.0% 100.0% 0.1 0.0% Objecter::_send_op
0.0 0.0% 100.0% 0.8 0.2% Objecter::handle_osd_op_reply
0.0 0.0% 100.0% 0.8 0.2% Objecter::ms_dispatch
0.0 0.0% 100.0% 0.8 0.2% Objecter::ms_fast_dispatch
0.0 0.0% 100.0% 0.1 0.0% Objecter::op_submit
0.0 0.0% 100.0% 0.0 0.0% Objecter::sg_write_trunc
0.0 0.0% 100.0% 0.0 0.0% OpHistory::insert
0.0 0.0% 100.0% 0.0 0.0% OpTracker::unregister_inflight_op
0.0 0.0% 100.0% 0.0 0.0% PrebufferedStreambuf::overflow
0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_after
0.0 0.0% 100.0% 0.0 0.0% SafeTimer::add_event_at
0.0 0.0% 100.0% 0.4 0.1% SafeTimer::timer_thread
0.0 0.0% 100.0% 0.4 0.1% SafeTimerThread::entry
0.0 0.0% 100.0% 0.0 0.0% Server::_session_logged
0.0 0.0% 100.0% 0.0 0.0% Server::apply_allocated_inos
0.0 0.0% 100.0% 1.1 0.3% Server::dispatch
0.0 0.0% 100.0% 1.7 0.4% Server::dispatch_client_request
0.0 0.0% 100.0% 0.7 0.2% Server::handle_client_getattr
0.0 0.0% 100.0% 0.9 0.2% Server::handle_client_open
0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_openc
0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_readdir
0.0 0.0% 100.0% 1.1 0.3% Server::handle_client_request
0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_session
0.0 0.0% 100.0% 0.0 0.0% Server::handle_client_setattr
0.0 0.0% 100.0% 0.0 0.0% Server::journal_and_reply
0.0 0.0% 100.0% 0.0 0.0% Server::journal_close_session
0.0 0.0% 100.0% 0.3 0.1% Server::rdlock_path_pin_ref
0.0 0.0% 100.0% 0.0 0.0% Server::recall_client_state
0.0 0.0% 100.0% 0.5 0.1% Server::reply_client_request
0.0 0.0% 100.0% 0.5 0.1% Server::respond_to_request
0.0 0.0% 100.0% 0.4 0.1% Server::set_trace_dist
0.0 0.0% 100.0% 0.0 0.0% SessionMap::_mark_dirty
0.0 0.0% 100.0% 0.0 0.0% SessionMap::mark_dirty
0.0 0.0% 100.0% 0.0 0.0% SessionMap::remove_session
0.0 0.0% 100.0% 0.0 0.0%
ceph::buffer::list::iterator_impl::copy
0.0 0.0% 100.0% 0.0 0.0%
ceph::buffer::list::iterator_impl::copy_shallow
0.0 0.0% 100.0% 400.0 100.0% clone
0.0 0.0% 100.0% 0.0 0.0% filepath::parse_bits
0.0 0.0% 100.0% 0.0 0.0% inode_t::operator=
0.0 0.0% 100.0% 0.0 0.0% operator<<@2a2890
0.0 0.0% 100.0% 0.0 0.0% operator<<@2c9760
0.0 0.0% 100.0% 0.0 0.0% operator<<@3eadf0
0.0 0.0% 100.0% 400.0 100.0% start_thread
0.0 0.0% 100.0% 0.1 0.0% std::__cxx11::basic_string::_M_append
0.0 0.0% 100.0% 0.0 0.0%
std::__cxx11::basic_string::_M_replace_aux
0.0 0.0% 100.0% 0.0 0.0% std::__cxx11::list::operator=
0.0 0.0% 100.0% 0.0 0.0% std::__ostream_insert
0.0 0.0% 100.0% 0.0 0.0% std::basic_streambuf::xsputn
0.0 0.0% 100.0% 0.0 0.0% std::num_put::_M_insert_int
0.0 0.0% 100.0% 0.0 0.0% std::num_put::do_put
0.0 0.0% 100.0% 0.0 0.0% std::operator<<
0.0 0.0% 100.0% 0.0 0.0% std::ostream::_M_insert
0.0 0.0% 100.0% 365.6 91.4% std::this_thread::__sleep_for
0.0 0.0% 100.0% 0.0 0.0% utime_t::localtime
0.0 0.0% 100.0% 0.1 0.0% void finish_contexts@2a30f0
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Greetings!!
2018-07-24 12:07 GMT+02:00 Yan, Zheng <[email protected]>:
> On Tue, Jul 24, 2018 at 4:59 PM Daniel Carrasco <[email protected]>
> wrote:
> >
> > Hello,
> >
> > How many time is neccesary?, because is a production environment and
> memory profiler + low cache size because the problem, gives a lot of CPU
> usage from OSD and MDS that makes it fails while profiler is running. Is
> there any problem if is done in a low traffic time? (less usage and maybe
> it don't fails, but maybe less info about usage).
> >
>
> just one time, wait a few minutes between start_profiler and stop_profiler
>
> > Greetings!
> >
> > 2018-07-24 10:21 GMT+02:00 Yan, Zheng <[email protected]>:
> >>
> >> I mean:
> >>
> >> ceph tell mds.x heap start_profiler
> >>
> >> ... wait for some time
> >>
> >> ceph tell mds.x heap stop_profiler
> >>
> >> pprof --text /usr/bin/ceph-mds
> >> /var/log/ceph/ceph-mds.x.profile.<largest number>.heap
> >>
> >>
> >>
> >>
> >> On Tue, Jul 24, 2018 at 3:18 PM Daniel Carrasco <[email protected]>
> wrote:
> >> >
> >> > This is what i get:
> >> >
> >> > --------------------------------------------------------
> >> > --------------------------------------------------------
> >> > --------------------------------------------------------
> >> > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap dump
> >> > 2018-07-24 09:05:19.350720 7fc562ffd700 0 client.1452545
> ms_handle_reset on 10.22.0.168:6800/1685786126
> >> > 2018-07-24 09:05:29.103903 7fc563fff700 0 client.1452548
> ms_handle_reset on 10.22.0.168:6800/1685786126
> >> > mds.kavehome-mgto-pro-fs01 dumping heap profile now.
> >> > ------------------------------------------------
> >> > MALLOC: 760199640 ( 725.0 MiB) Bytes in use by application
> >> > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> >> > MALLOC: + 246962320 ( 235.5 MiB) Bytes in central cache freelist
> >> > MALLOC: + 43933664 ( 41.9 MiB) Bytes in transfer cache freelist
> >> > MALLOC: + 41012664 ( 39.1 MiB) Bytes in thread cache freelists
> >> > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata
> >> > MALLOC: ------------
> >> > MALLOC: = 1102295200 ( 1051.2 MiB) Actual memory used (physical +
> swap)
> >> > MALLOC: + 4268335104 ( 4070.6 MiB) Bytes released to OS (aka
> unmapped)
> >> > MALLOC: ------------
> >> > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used
> >> > MALLOC:
> >> > MALLOC: 33027 Spans in use
> >> > MALLOC: 19 Thread heaps in use
> >> > MALLOC: 8192 Tcmalloc page size
> >> > ------------------------------------------------
> >> > Call ReleaseFreeMemory() to release freelist memory to the OS (via
> madvise()).
> >> > Bytes released to the OS take up virtual address space but no
> physical memory.
> >> >
> >> >
> >> > --------------------------------------------------------
> >> > --------------------------------------------------------
> >> > --------------------------------------------------------
> >> > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats
> >> > 2018-07-24 09:14:25.747706 7f94fffff700 0 client.1452578
> ms_handle_reset on 10.22.0.168:6800/1685786126
> >> > 2018-07-24 09:14:25.754034 7f95057fa700 0 client.1452581
> ms_handle_reset on 10.22.0.168:6800/1685786126
> >> > mds.kavehome-mgto-pro-fs01 tcmalloc heap
> stats:------------------------------------------------
> >> > MALLOC: 960649328 ( 916.1 MiB) Bytes in use by application
> >> > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> >> > MALLOC: + 108867288 ( 103.8 MiB) Bytes in central cache freelist
> >> > MALLOC: + 37179424 ( 35.5 MiB) Bytes in transfer cache freelist
> >> > MALLOC: + 40143000 ( 38.3 MiB) Bytes in thread cache freelists
> >> > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata
> >> > MALLOC: ------------
> >> > MALLOC: = 1157025952 ( 1103.4 MiB) Actual memory used (physical +
> swap)
> >> > MALLOC: + 4213604352 ( 4018.4 MiB) Bytes released to OS (aka
> unmapped)
> >> > MALLOC: ------------
> >> > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used
> >> > MALLOC:
> >> > MALLOC: 33028 Spans in use
> >> > MALLOC: 19 Thread heaps in use
> >> > MALLOC: 8192 Tcmalloc page size
> >> > ------------------------------------------------
> >> > Call ReleaseFreeMemory() to release freelist memory to the OS (via
> madvise()).
> >> > Bytes released to the OS take up virtual address space but no
> physical memory.
> >> >
> >> > --------------------------------------------------------
> >> > --------------------------------------------------------
> >> > --------------------------------------------------------
> >> > After heap release:
> >> > :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats
> >> > 2018-07-24 09:15:28.540203 7f2f7affd700 0 client.1443339
> ms_handle_reset on 10.22.0.168:6800/1685786126
> >> > 2018-07-24 09:15:28.547153 7f2f7bfff700 0 client.1443342
> ms_handle_reset on 10.22.0.168:6800/1685786126
> >> > mds.kavehome-mgto-pro-fs01 tcmalloc heap
> stats:------------------------------------------------
> >> > MALLOC: 710315776 ( 677.4 MiB) Bytes in use by application
> >> > MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist
> >> > MALLOC: + 246471880 ( 235.1 MiB) Bytes in central cache freelist
> >> > MALLOC: + 40802848 ( 38.9 MiB) Bytes in transfer cache freelist
> >> > MALLOC: + 38689304 ( 36.9 MiB) Bytes in thread cache freelists
> >> > MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata
> >> > MALLOC: ------------
> >> > MALLOC: = 1046466720 ( 998.0 MiB) Actual memory used (physical +
> swap)
> >> > MALLOC: + 4324163584 ( 4123.8 MiB) Bytes released to OS (aka
> unmapped)
> >> > MALLOC: ------------
> >> > MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used
> >> > MALLOC:
> >> > MALLOC: 33177 Spans in use
> >> > MALLOC: 19 Thread heaps in use
> >> > MALLOC: 8192 Tcmalloc page size
> >> > ------------------------------------------------
> >> > Call ReleaseFreeMemory() to release freelist memory to the OS (via
> madvise()).
> >> > Bytes released to the OS take up virtual address space but no
> physical memory.
> >> >
> >> >
> >> > The other commands fails with a curl error:
> >> > Failed to get profile: curl 'http:///pprof/profile?seconds=30' >
> /root/pprof/.tmp.ceph-mds.1532416424.:
> >> >
> >> >
> >> > Greetings!!
> >> >
> >> > 2018-07-24 5:35 GMT+02:00 Yan, Zheng <[email protected]>:
> >> >>
> >> >> could you profile memory allocation of mds
> >> >>
> >> >> http://docs.ceph.com/docs/mimic/rados/troubleshooting/
> memory-profiling/
> >> >> On Tue, Jul 24, 2018 at 7:54 AM Daniel Carrasco <
> [email protected]> wrote:
> >> >> >
> >> >> > Yeah, is also my thread. This thread was created before lower the
> cache size from 512Mb to 8Mb. I thought that maybe was my fault and I did a
> misconfiguration, so I've ignored the problem until now.
> >> >> >
> >> >> > Greetings!
> >> >> >
> >> >> > El mar., 24 jul. 2018 1:00, Gregory Farnum <[email protected]>
> escribió:
> >> >> >>
> >> >> >> On Mon, Jul 23, 2018 at 11:08 AM Patrick Donnelly <
> [email protected]> wrote:
> >> >> >>>
> >> >> >>> On Mon, Jul 23, 2018 at 5:48 AM, Daniel Carrasco <
> [email protected]> wrote:
> >> >> >>> > Hi, thanks for your response.
> >> >> >>> >
> >> >> >>> > Clients are about 6, and 4 of them are the most of time on
> standby. Only two
> >> >> >>> > are active servers that are serving the webpage. Also we've a
> varnish on
> >> >> >>> > front, so are not getting all the load (below 30% in PHP is
> not much).
> >> >> >>> > About the MDS cache, now I've the mds_cache_memory_limit at
> 8Mb.
> >> >> >>>
> >> >> >>> What! Please post `ceph daemon mds.<name> config diff`, `...
> perf
> >> >> >>> dump`, and `... dump_mempools ` from the server the active MDS
> is on.
> >> >> >>>
> >> >> >>> > I've tested
> >> >> >>> > also 512Mb, but the CPU usage is the same and the MDS RAM
> usage grows up to
> >> >> >>> > 15GB (on a 16Gb server it starts to swap and all fails). With
> 8Mb, at least
> >> >> >>> > the memory usage is stable on less than 6Gb (now is using
> about 1GB of RAM).
> >> >> >>>
> >> >> >>> We've seen reports of possible memory leaks before and the
> potential
> >> >> >>> fixes for those were in 12.2.6. How fast does your MDS reach
> 15GB?
> >> >> >>> Your MDS cache size should be configured to 1-8GB (depending on
> your
> >> >> >>> preference) so it's disturbing to see you set it so low.
> >> >> >>
> >> >> >>
> >> >> >> See also the thread "[ceph-users] Fwd: MDS memory usage is very
> high", which had more discussion of that. The MDS daemon seemingly had
> 9.5GB of allocated RSS but only believed 489MB was in use for the cache...
> >> >> >> -Greg
> >> >> >>
> >> >> >>>
> >> >> >>>
> >> >> >>> --
> >> >> >>> Patrick Donnelly
> >> >> >>> _______________________________________________
> >> >> >>> ceph-users mailing list
> >> >> >>> [email protected]
> >> >> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >> >
> >> >> > _______________________________________________
> >> >> > ceph-users mailing list
> >> >> > [email protected]
> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > _________________________________________
> >> >
> >> > Daniel Carrasco Marín
> >> > Ingeniería para la Innovación i2TIC, S.L.
> >> > Tlf: +34 911 12 32 84 Ext: 223
> >> > www.i2tic.com
> >> > _________________________________________
> >
> >
> >
> >
> > --
> > _________________________________________
> >
> > Daniel Carrasco Marín
> > Ingeniería para la Innovación i2TIC, S.L.
> > Tlf: +34 911 12 32 84 Ext: 223
> > www.i2tic.com
> > _________________________________________
>
--
_________________________________________
Daniel Carrasco Marín
Ingeniería para la Innovación i2TIC, S.L.
Tlf: +34 911 12 32 84 Ext: 223
www.i2tic.com
_________________________________________
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
