Hi all,
I'm experiencing random interface hangs (load indicator stays on) as
if WT loops are not ending.
I have attached to it with a gdb and this is what I'm seeing:
1) Some threads waiting to acquire a lock for complete an
Wt::WebController::handleApplicationEvent
Like this one:
Thread 6 (Thread 0x7fa7237fe700 (LWP 12681)):
#0 __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x00007fa7435035d9 in _L_lock_953 () from /lib/libpthread.so.0
#2 0x00007fa7435033fb in __pthread_mutex_lock (mutex=0x7fa7241eb368)
at pthread_mutex_lock.c:61
#3 0x00007fa7544986c9 in boost::mutex::lock (this=0x7fa7237fda40,
session=..., takeLock=true) at
/usr/include/boost/thread/pthread/mutex.hpp:50
#4 boost::unique_lock<boost::mutex>::lock (this=0x7fa7237fda40,
session=..., takeLock=true) at /usr/include/boost/thread/locks.hpp:338
#5 Handler (this=0x7fa7237fda40, session=..., takeLock=true) at
/root/Installed/wt-3.2.1-patched/src/web/WebSession.C:690
#6 0x00007fa754489ac3 in Wt::WebController::handleApplicationEvent
(this=<value optimized out>, event=...) at
/root/Installed/wt-3.2.1-patched/src/web/WebController.C:426
#7 0x00007fa75428e62e in boost::function0<void>::operator()() const
() from /usr/local/lib/libwt.so.32
#8 0x00007fa75428e8ca in
boost::asio::detail::handler_queue::handler_wrapper<boost::function<void
()()> >::do_call(boost::asio::detail::handler_queue::handler*) () from
/usr/local/lib/libwt.so.32
#9 0x00007fa757c3da18 in
boost::asio::detail::handler_queue::handler::invoke (this=0x10351820,
ec=<value optimized out>) at
/usr/include/boost/asio/detail/handler_queue.hpp:39
#10
boost::asio::detail::task_io_service<boost::asio::detail::epoll_reactor<false>
>::do_one (this=0x10351820, ec=<value optimized out>) at
/usr/include/boost/asio/detail/task_io_service.hpp:268
#11
boost::asio::detail::task_io_service<boost::asio::detail::epoll_reactor<false>
>::run (this=0x10351820, ec=<value optimized out>) at
/usr/include/boost/asio/detail/task_io_service.hpp:103
#12 0x00007fa754289c44 in boost::asio::io_service::run
(this=0x103511c0) at /usr/include/boost/asio/impl/io_service.ipp:58
---Type <return> to continue, or q <return> to quit---
#13 Wt::WIOService::run (this=0x103511c0) at
/root/Installed/wt-3.2.1-patched/src/Wt/WIOService.C:128
#14 0x00007fa75067eb70 in thread_proxy () from
/usr/lib/libboost_thread.so.1.40.0
#15 0x00007fa7435009ca in start_thread (arg=<value optimized out>) at
pthread_create.c:300
#16 0x00007fa74325d70d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#17 0x0000000000000000 in ?? ()
the mutex=0x7fa7241eb368 is owned by thread 12678:
(gdb) p *mutex
$5 = {__data = {__lock = 2, __count = 0, __owner = 12678, __nusers =
1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = "\002\000\000\000\000\000\000\000\206\061\000\000\001",
'\000' <repeats 26 times>,
__align = 2}
and that thread is indeed a thread executing an handleApplicationEvent
and it's destroying a WAbstractItemModel:
Thread 9 (Thread 0x7fa720bcb700 (LWP 12678)):
#0 __lll_lock_wait_private () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:97
#1 0x00007fa7431f63dc in _L_lock_12449 () from /lib/libc.so.6
#2 0x00007fa7431f4e78 in *__GI___libc_free (mem=0x7fa710000020) at
malloc.c:3736
#3 0x00007fa750897bfc in
boost::signals::detail::signal_base_impl::~signal_base_impl() () from
/usr/lib/libboost_signals.so.1.40.0
#4 0x00007fa7508985b2 in
boost::detail::sp_counted_impl_p<boost::signals::detail::signal_base_impl>::dispose()
() from /usr/lib/libboost_signals.so.1.40.0
#5 0x00007fa750897d99 in
boost::signals::detail::signal_base::~signal_base() () from
/usr/lib/libboost_signals.so.1.40.0
#6 0x00007fa7584dff8e in ~signal6 (this=0x7fa7050bcc60,
__in_chrg=<value optimized out>) at
/usr/include/boost/signals/signal_template.hpp:142
#7 ~Signal (this=0x7fa7050bcc60, __in_chrg=<value optimized out>) at
/usr/local/include/Wt/WSignal:767
#8 0x00007fa75419a9e3 in ~WAbstractItemModel (this=0x7fa7050bca60,
__in_chrg=<value optimized out>) at
/root/Installed/wt-3.2.1-patched/src/Wt/WAbstractItemModel.C:41
#9 0x00007fa75432764f in ~WStandardItemModel (this=0x7fa7050bca60,
__in_chrg=<value optimized out>) at
/root/Installed/wt-3.2.1-patched/src/Wt/WStandardItemModel.C:45
#10 0x000000000041c669 in boost::detail::sp_counted_base::release
(this=<value optimized out>, __in_chrg=<value optimized out>) at
/usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
#11 ~shared_count (this=<value optimized out>, __in_chrg=<value
optimized out>) at
/usr/include/boost/smart_ptr/detail/shared_count.hpp:217
#12 0x00007fa756aa5d2c in ~shared_ptr (this=0x7fa710757080,
aDataModelBuffer=DWARF-2 expression error: DW_OP_reg operations must
be used either alone or in conjuction with DW_OP_piece.
) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
#13 boost::shared_ptr<Wt::WStandardItemModel>::operator=
(this=0x7fa710757080, aDataModelBuffer=DWARF-2 expression error:
DW_OP_reg operations must be used either alone or in conjuction with
DW_OP_piece.
) at /usr/include/boost/smart_ptr/shared_ptr.hpp:305
#14 plots::WtTerminal::ChartOwner::swapModel (this=0x7fa710757080,
aDataModelBuffer=DWARF-2 expression error: DW_OP_reg operations must
be used either alone or in conjuction with DW_OP_piece.
) at /home/essa/3.8.0.0/src/Libraries/Plots/wtterminal.cpp:151
#15 0x00007fa756aa7a5b in
call<boost::shared_ptr<plots::WtTerminal::ChartOwner>,
boost::shared_ptr<Wt::WStandardItemModel>, std::map<unsigned int,
boost::tuples::tuple<plots::LineStyle, plots::PointType, plots::Color,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type>, std::less<unsigned int>,
std::allocator<std::pair<unsigned int const,
boost::tuples::tuple<plots::LineStyle, plots::PointType, plots::Color,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type> > > > const,
boost::shared_ptr<plots::GenericWtAxisRange>,
boost::shared_ptr<plots::GenericWtAxisRange> >
(function_obj_ptr=<value optimized out>, a0=<value optimized out>)
at /usr/include/boost/bind/mem_fn_template.hpp:485
#16 operator()<boost::shared_ptr<plots::WtTerminal::ChartOwner> >
(function_obj_ptr=<value optimized out>, a0=<value optimized out>) at
/usr/include/boost/bind/mem_fn_template.hpp:499
#17 operator()<boost::_mfi::mf4<void, plots::WtTerminal::ChartOwner,
boost::shared_ptr<Wt::WStandardItemModel>, const std::map<unsigned
int, boost::tuples::tuple<plots::LineStyle, plots::PointType,
plots::Color, boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type>, std::less<unsigned int>,
std::allocator<std::pair<const unsigned int,
boost::tuples::tuple<plots::LineStyle, plots::PointType, plots::Color,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type> > > >&,
boost::shared_ptr<plots::GenericWtAxisRange>,
boost::shared_ptr<plots::GenericWtAxisRange> >,
boost::_bi::list1<boost::shared_ptr<plots::WtTerminal::ChartOwner>&> >
(function_obj_ptr=<value optimized out>,
a0=<value optimized out>) at /usr/include/boost/bind/bind.hpp:518
#18 operator()<boost::shared_ptr<plots::WtTerminal::ChartOwner> >
(function_obj_ptr=<value optimized out>, a0=<value optimized out>) at
/usr/include/boost/bind/bind_template.hpp:32
#19 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void,
boost::_mfi::mf4<void, plots::WtTerminal::ChartOwner,
boost::shared_ptr<Wt::WStandardItemModel>, std::map<unsigned int,
boost::tuples::tuple<plots::LineStyle, plots::PointType, plots::Color,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type>, std::less<unsigned int>,
std::allocator<std::pair<unsigned int const,
boost::tuples::tuple<plots::LineStyle, plots::PointType, plots::Color,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type> > > > const&,
boost::shared_ptr<plots::GenericWtAxisRange>,
boost::shared_ptr<plots::GenericWtAxisRange> >,
boost::_bi::list5<boost::arg<1>,
boost::_bi::value<boost::shared_ptr<Wt::WStandardItemModel> >,
boost::_bi::value<std::map<unsigned int,
boost::tuples::tuple<plots::LineStyle, plots::PointType, plots::Color,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type>, std::less<unsigned int>,
std::allocator<std::pair<unsigned int const,
boost::tuples::tuple<plots::LineStyle, plots::PointType,
plots::Color---Type <return> to continue, or q <return> to quit---
, boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type, boost::tuples::null_type,
boost::tuples::null_type> > > > >,
boost::_bi::value<boost::shared_ptr<plots::WtLinearRange> >,
boost::_bi::value<boost::shared_ptr<plots::WtLinearRange> > > >, void,
boost::shared_ptr<plots::WtTerminal::ChartOwner> >::invoke
(function_obj_ptr=<value optimized out>, a0=<value optimized out>)
at /usr/include/boost/function/function_template.hpp:153
#20 0x00007fa756ab1dbf in boost::function1<void,
boost::shared_ptr<plots::WtTerminal::ChartOwner> >::operator()
(function_obj_ptr=<value optimized out>) at
/usr/include/boost/function/function_template.hpp:1013
#21
operator()<boost::function<void(boost::shared_ptr<plots::WtTerminal::ChartOwner>)>,
boost::_bi::list0> (function_obj_ptr=<value optimized out>) at
/usr/include/boost/bind/bind.hpp:246
#22 boost::_bi::bind_t<boost::_bi::unspecified, boost::function<void
()(boost::shared_ptr<plots::WtTerminal::ChartOwner>)>,
boost::_bi::list1<boost::_bi::value<boost::shared_ptr<plots::WtTerminal::ChartOwner>
> > >::operator()() (
function_obj_ptr=<value optimized out>) at
/usr/include/boost/bind/bind_template.hpp:20
#23
boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<boost::_bi::unspecified,
boost::function<void
()(boost::shared_ptr<plots::WtTerminal::ChartOwner>)>,
boost::_bi::list1<boost::_bi::value<boost::shared_ptr<plots::WtTerminal::ChartOwner>
> > >, void>::invoke(boost::detail::function::function_buffer&)
(function_obj_ptr=<value optimized out>) at
/usr/include/boost/function/function_template.hpp:153
#24 0x00007fa7544a39f6 in boost::function0<void>::operator()
(this=0x7fa7241eb2d0, event=...) at
/usr/include/boost/function/function_template.hpp:1013
#25 Wt::WebSession::notify (this=0x7fa7241eb2d0, event=...) at
/root/Installed/wt-3.2.1-patched/src/web/WebSession.C:1712
#26 0x00007fa754489b90 in Wt::WebController::handleApplicationEvent
(this=<value optimized out>, event=...) at
/root/Installed/wt-3.2.1-patched/src/web/WebController.C:430
#27 0x00007fa75428e62e in boost::function0<void>::operator()() const
() from /usr/local/lib/libwt.so.32
#28 0x00007fa75428e8ca in
boost::asio::detail::handler_queue::handler_wrapper<boost::function<void
()()> >::do_call(boost::asio::detail::handler_queue::handler*) () from
/usr/local/lib/libwt.so.32
#29 0x00007fa757c3da18 in
boost::asio::detail::handler_queue::handler::invoke (this=0x10351820,
ec=<value optimized out>) at
/usr/include/boost/asio/detail/handler_queue.hpp:39
#30
boost::asio::detail::task_io_service<boost::asio::detail::epoll_reactor<false>
>::do_one (this=0x10351820, ec=<value optimized out>) at
/usr/include/boost/asio/detail/task_io_service.hpp:268
#31
boost::asio::detail::task_io_service<boost::asio::detail::epoll_reactor<false>
>::run (this=0x10351820, ec=<value optimized out>) at
/usr/include/boost/asio/detail/task_io_service.hpp:103
#32 0x00007fa754289c44 in boost::asio::io_service::run
(this=0x103511c0) at /usr/include/boost/asio/impl/io_service.ipp:58
#33 Wt::WIOService::run (this=0x103511c0) at
/root/Installed/wt-3.2.1-patched/src/Wt/WIOService.C:128
#34 0x00007fa75067eb70 in thread_proxy () from
/usr/lib/libboost_thread.so.1.40.0
#35 0x00007fa7435009ca in start_thread (arg=<value optimized out>) at
pthread_create.c:300
#36 0x00007fa74325d70d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#37 0x0000000000000000 in ?? ()
2) One of our threads waiting to for an updatelock:
Thread 3 (Thread 0x7fa71bc09700 (LWP 17263)):
#0 0x00007fa7435033d4 in __pthread_mutex_lock (mutex=0x7fa7241eb2e0)
at pthread_mutex_lock.c:51
#1 0x00007fa7541e3101 in
Wt::UpdateLockImpl::UpdateLockImpl(Wt::WApplication*) () from
/usr/local/lib/libwt.so.32
#2 0x00007fa7541d6ab6 in UpdateLock (this=0x7fa71bc08b90,
app=0x7fa7103afce0) at
/root/Installed/wt-3.2.1-patched/src/Wt/WApplication.C:1424
#3 0x00007fa754bd686f in EssaWebInterface::onStatusChange
(this=0x7fa7103afce0, aApplicationName=...) at
/home/essa/3.8.0.0/src/Libraries/WebInterfaceLib/essawebinterface.cpp:123
#4 0x00007fa754bd6f47 in EssaWebInterface::updateFromRunner
(this=0x7fa7103afce0, aRunner=DWARF-2 expression error: DW_OP_reg
operations must be used either alone or in conjuction with
DW_OP_piece.
) at /home/essa/3.8.0.0/src/Libraries/WebInterfaceLib/essawebinterface.cpp:87
#5 0x00007fa754bd6ff4 in EssaWebInterface::timerLoop
(this=0x7fa7103afce0) at
/home/essa/3.8.0.0/src/Libraries/WebInterfaceLib/essawebinterface.cpp:81
#6 0x00007fa754bded96 in boost::function0<void>::operator()
(this=0x7fa7103b0318) at
/usr/include/boost/function/function_template.hpp:1013
#7 support::ThreadStatusUpdater::singleRun (this=0x7fa7103b0318) at
/home/essa/3.8.0.0/project/linux/cmake/../../../src/Libraries/Support/threadstatusupdater.h:51
#8 0x00007fa754ee8549 in Thread::operator()() () from
/opt/essa-suite/lib/libsupport.so
#9 0x00007fa75067eb70 in thread_proxy () from
/usr/lib/libboost_thread.so.1.40.0
#10 0x00007fa7435009ca in start_thread (arg=<value optimized out>) at
pthread_create.c:300
#11 0x00007fa74325d70d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
---Type <return> to continue, or q <return> to quit---
#12 0x0000000000000000 in ?? ()
now if I issue a continue command in gdb, I get:
(gdb) continue
Continuing.
[Thread 0x7fa71bc09700 (LWP 17263) exited] <== This is the thread
trying to acquire a UpdateLockImpl
so as you can see it was able to attach with the gdb, issue a continue
command and the hang disappeared.
I'm running out of ideas.
--
cpp-today.blogspot.com
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
witty-interest mailing list
witty-interest@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/witty-interest
