Hi Doug,
I know this command, you've taught me many times :)
I did examine other threads, but find nothing unusual. Full story followed:
$ gdb -c core.333 hypertable/0.9.0.10/bin/Hypertable.RangeServer
[...]
Core was generated by
`/home/ht4/hypertable/0.9.0.10/bin/Hypertable.RangeServer
--pidfile=/home/ht4/hy'.
Program terminated with signal 11, Segmentation fault.
[...]
#0 0x000000302af71900 in memcpy () from /lib64/tls/libc.so.6
(gdb) thread apply all where
Thread 30 (process 333):
#0 0x000000302b806ffb in pthread_join () from /lib64/tls/libpthread.so.0
#1 0x0000002a95b8a1be in boost::thread::join () from
/home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#2 0x0000002a95b8a336 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#3 0x000000000051b5f8 in Hypertable::ApplicationQueue::join (this=0xa6bdc0)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:189
#4 0x000000000051ae22 in main (argc=3, argv=0x7fbffffba8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/main.cc:171
Thread 29 (process 341):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
(this=0x40a001a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
#2 0x000000000062c1e3 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (
[EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#3 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 28 (process 342):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
(this=0x414011a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
#2 0x000000000062c1e3 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (
[EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#3 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 27 (process 343):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
(this=0x41e021a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
#2 0x000000000062c1e3 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (
[EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#3 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 26 (process 344):
#0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
(this=0x428031a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
#2 0x000000000062c1e3 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
void>::invoke (
[EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#3 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#4 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 25 (process 345):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6af30, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6af30,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x0000000000622285 in Hypertable::ConnectionManager::operator()
(this=0x432041a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ConnectionManager.cc:299
#4 0x000000000051f443 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ConnectionManager,
void>::invoke (
[EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 24 (process 346):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x43c051a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 23 (process 347):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x446061a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 22 (process 348):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x450071a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 21 (process 349):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x45a081a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 20 (process 350):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x464091a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 19 (process 351):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x46e0a1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 18 (process 352):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4780b1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 17 (process 353):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4820c1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 16 (process 354):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x48c0d1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 15 (process 355):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4960e1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 14 (process 356):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4a00f1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 13 (process 357):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4aa101a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 12 (process 358):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4b4111a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 11 (process 359):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4be121a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 10 (process 360):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4c8131a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 9 (process 361):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4d2141a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 8 (process 362):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4dc151a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 7 (process 363):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4e6161a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 6 (process 364):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4f0171a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 5 (process 365):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0xa6be80, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0xa6be80,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x000000000051e4c0 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x4fa181a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
#4 0x000000000051e441 in
boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#5 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#6 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#9 0x0000000000000000 in ?? ()
Thread 4 (process 366):
#0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
(this=0x50418008, [EMAIL PROTECTED])
at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
#2 0x000000000051eacb in
boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
> (this=0x50418008,
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
#3 0x00000000006175ac in
Hypertable::DispatchHandlerSynchronizer::wait_for_reply
(this=0x50417f80, [EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/DispatchHandlerSynchronizer.cc:59
#4 0x000000000058f20d in Hypertable::DfsBroker::Client::close
(this=0xa6fe10, fd=82320)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/DfsBroker/Lib/Client.cc:224
#5 0x000000000057b3b9 in ~CellStoreV0 (this=0x3829c00)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/CellStoreV0.cc:66
#6 0x000000000051c8e1 in Hypertable::intrusive_ptr_release (rc=0x3829c00)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/ReferenceCount.h:73
#7 0x000000000055f57a in ~intrusive_ptr (this=0x113f5908) at
/usr/local/include/boost-1_34_1/boost/intrusive_ptr.hpp:83
#8 0x000000000058ac6f in ~CellStoreScannerV0 (this=0x113f58f0)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/CellStoreScannerV0.cc:232
#9 0x00000000005814f3 in ~MergeScanner (this=0x2ac9208000)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MergeScanner.cc:53
#10 0x000000000051c8e1 in Hypertable::intrusive_ptr_release (rc=0x2ac9208000)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/ReferenceCount.h:73
#11 0x0000000000524e34 in ~intrusive_ptr (this=0x50418ef0) at
/usr/local/include/boost-1_34_1/boost/intrusive_ptr.hpp:83
#12 0x000000000056e00c in Hypertable::AccessGroup::run_compaction
(this=0xa7a7c00, timestamp=
{logical = 1222151526556990005, real = 1222151526557728000}, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:430
#13 0x000000000055d5dc in Hypertable::Range::run_compaction
(this=0x2bda9900, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
#14 0x000000000055d4e9 in Hypertable::Range::compact (this=0x2bda9900,
major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
#15 0x0000000000556ea9 in
Hypertable::MaintenanceTaskCompaction::execute (this=0x2ab7fe88c0)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
#16 0x0000000000541a4f in
Hypertable::MaintenanceQueue::Worker::operator() (this=0x504191a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
#17 0x0000000000541885 in
boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#18 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#19 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#20 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#21 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#22 0x0000000000000000 in ?? ()
Thread 3 (process 367):
#0 0x000000302b808b9f in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/tls/libpthread.so.0
#1 0x0000002a95b8886c in boost::detail::condition_impl::do_timed_wait ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#2 0x0000000000541dee in
boost::condition::do_timed_wait<boost::mutex> (this=0x2aeabf3e48,
[EMAIL PROTECTED],
[EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:178
#3 0x0000000000541da9 in
boost::condition::timed_wait<boost::detail::thread::scoped_lock<boost::mutex>
> (
this=0x2aeabf3e48, [EMAIL PROTECTED], [EMAIL PROTECTED]) at
/usr/local/include/boost-1_34_1/boost/thread/condition.hpp:112
#4 0x00000000005d04e2 in
Hypertable::TableMutatorCompletionCounter::wait_for_completion
(this=0x2aeabf3e20,
[EMAIL PROTECTED]) at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:64
#5 0x00000000005cf2b8 in
Hypertable::TableMutatorScatterBuffer::wait_for_completion
(this=0x2aeabf3d80, [EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:255
#6 0x00000000005ccaef in
Hypertable::TableMutator::wait_for_previous_buffer (this=0x2af826e0c0,
[EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutator.cc:253
#7 0x00000000005cc917 in Hypertable::TableMutator::flush (this=0x2af826e0c0)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutator.cc:206
#8 0x0000000000585708 in Hypertable::MetadataNormal::write_files
(this=0x50e19c50, [EMAIL PROTECTED],
[EMAIL PROTECTED]) at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MetadataNormal.cc:101
#9 0x000000000056f0e5 in Hypertable::AccessGroup::update_files_column
(this=0x2ac64ca800)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:571
#10 0x000000000056de6f in Hypertable::AccessGroup::run_compaction
(this=0x2ac64ca800, timestamp=
{logical = 1222151526557045002, real = 1222151526557728000}, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:422
#11 0x000000000055d5dc in Hypertable::Range::run_compaction
(this=0x9868480, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
#12 0x000000000055d4e9 in Hypertable::Range::compact (this=0x9868480,
major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
#13 0x0000000000556ea9 in
Hypertable::MaintenanceTaskCompaction::execute (this=0x3bf883f0)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
#14 0x0000000000541a4f in
Hypertable::MaintenanceQueue::Worker::operator() (this=0x50e1a1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
#15 0x0000000000541885 in
boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#16 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#17 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#18 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#19 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#20 0x0000000000000000 in ?? ()
Thread 2 (process 368):
#0 0x00000000005f8132 in _lzo1x_1_do_compress (in=0x2ae664c000
"\034e9d31a63d844791c", in_len=264505, out=0x2afb56701a "",
out_len=0x5181a568, wrkmem=0x2ac4607000)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/lzo/minilzo.c:2713
#1 0x00000000005f82fb in lzo1x_1_compress (in=0x2ae664c000
"\034e9d31a63d844791c", in_len=264505, out=0x2afb56701a "",
out_len=0x5181a568, wrkmem=0x2ac4607000)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/lzo/minilzo.c:2785
#2 0x00000000005dc9f6 in
Hypertable::BlockCompressionCodecLzo::deflate (this=0x29463840,
[EMAIL PROTECTED],
[EMAIL PROTECTED], [EMAIL PROTECTED], reserve=0)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/BlockCompressionCodecLzo.cc:74
#3 0x000000000057c61d in Hypertable::CellStoreV0::finalize
(this=0xa426e00, [EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/CellStoreV0.cc:211
#4 0x000000000056d70c in Hypertable::AccessGroup::run_compaction
(this=0xef02400, timestamp=
{logical = 1222151526557057001, real = 1222151526557728000}, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:361
#5 0x000000000055d5dc in Hypertable::Range::run_compaction
(this=0x1a8ab900, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
#6 0x000000000055d4e9 in Hypertable::Range::compact (this=0x1a8ab900,
major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
#7 0x0000000000556ea9 in
Hypertable::MaintenanceTaskCompaction::execute (this=0x171a6ab8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
#8 0x0000000000541a4f in
Hypertable::MaintenanceQueue::Worker::operator() (this=0x5181b1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
#9 0x0000000000541885 in
boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#10 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#11 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#12 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#13 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#14 0x0000000000000000 in ?? ()
Current language: auto; currently c
Thread 1 (process 369):
#0 0x000000302af71900 in memcpy () from /lib64/tls/libc.so.6
#1 0x00000000006354d8 in Hypertable::InetAddr::initialize
(addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/InetAddr.cc:71
#2 0x00000000005e855a in Hypertable::LocationCache::location_to_addr
(location=0x1311fd8 "10.65.25.163_38060",
[EMAIL PROTECTED]) at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/LocationCache.cc:256
#3 0x00000000005ce19c in Hypertable::TableMutatorScatterBuffer::set
(this=0xff4c900, [EMAIL PROTECTED], value=0x165e5c18,
value_len=639, [EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:72
#4 0x00000000005cc05b in Hypertable::TableMutator::set
(this=0x2ab21ec240, timestamp=0, [EMAIL PROTECTED], value=0x165e5c18,
value_len=639) at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutator.cc:93
#5 0x00000000005856f7 in Hypertable::MetadataNormal::write_files
(this=0x5221bc50, [EMAIL PROTECTED], [EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MetadataNormal.cc:100
#6 0x000000000056f0e5 in Hypertable::AccessGroup::update_files_column
(this=0x1bc2a00)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:571
#7 0x000000000056de6f in Hypertable::AccessGroup::run_compaction
(this=0x1bc2a00, timestamp=
{logical = 1222151526557033001, real = 1222151526557728000}, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:422
#8 0x000000000055d5dc in Hypertable::Range::run_compaction
(this=0x19a1000, major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
#9 0x000000000055d4e9 in Hypertable::Range::compact (this=0x19a1000,
major=false)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
#10 0x0000000000556ea9 in
Hypertable::MaintenanceTaskCompaction::execute (this=0x1036150)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
#11 0x0000000000541a4f in
Hypertable::MaintenanceQueue::Worker::operator() (this=0x5221c1a8)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
#12 0x0000000000541885 in
boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
void>::invoke ([EMAIL PROTECTED])
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
#13 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#14 0x0000002a95b8a407 in boost::thread_group::join_all ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#15 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#16 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#17 0x0000000000000000 in ?? ()
(gdb) thread 2
[Switching to thread 2 (process 368)]#0 0x00000000005f8132 in
_lzo1x_1_do_compress (
in=0x2ae664c000 "\034e9d31a63d844791c", in_len=264505,
out=0x2afb56701a "", out_len=0x5181a568, wrkmem=0x2ac4607000)
at
/home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/lzo/minilzo.c:2713
2713 while (ip < end && *m == *ip)
(gdb) p/x ip
$8 = 0x2ae665bcab
(gdb) p/x end
$9 = 0x2ae668c939
(gdb) p *ip
$10 = 9 '\t'
(gdb) p *m
$11 = 9 '\t'
Donald
On Tue, Sep 23, 2008 at 10:55 PM, Doug Judd <[EMAIL PROTECTED]> wrote:
> Hi Donald,
>
> The segfault may have happened in a different thread. To view the stack
> traces for all threads, issue this command:
>
> (gdb) thread apply all where
>
> You can then switch threads with the 'thread <tid>' command. Try examining
> the other threads to see if they were the cause.
>
> - Doug
>
> On Tue, Sep 23, 2008 at 6:01 AM, donald <[EMAIL PROTECTED]> wrote:
>>
>> Hi All,
>>
>> I meet this strange problem while loading 500GB log data into 7
>> Hypertable Range Servers, one range server dies with a core dump.
>> According to gdb output, the core is generated due to segmentation
>> fault in memcpy(). However, I can access both the source and
>> destination address in gdb:
>>
>> $ gdb -c core.333 Hypertable.RangeServer
>> [...]
>> (gdb) where
>> #0 0x000000302af71900 in memcpy () from /lib64/tls/libc.so.6
>> #1 0x00000000006354d8 in Hypertable::InetAddr::initialize
>> (addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/
>> InetAddr.cc:71
>> #2 0x00000000005e855a in Hypertable::LocationCache::location_to_addr
>> (location=0x1311fd8 "10.65.25.163_38060", [EMAIL PROTECTED])
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/
>> LocationCache.cc:256
>> #3 0x00000000005ce19c in Hypertable::TableMutatorScatterBuffer::set
>> (this=0xff4c900, [EMAIL PROTECTED], value=0x165e5c18,
>> value_len=639, [EMAIL PROTECTED])
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/
>> TableMutatorScatterBuffer.cc:72
>> #4 0x00000000005cc05b in Hypertable::TableMutator::set
>> (this=0x2ab21ec240, timestamp=0, [EMAIL PROTECTED], value=0x165e5c18,
>> value_len=639) at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/
>> Hypertable/Lib/TableMutator.cc:93
>> #5 0x00000000005856f7 in Hypertable::MetadataNormal::write_files
>> (this=0x5221bc50, [EMAIL PROTECTED], [EMAIL PROTECTED])
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>> RangeServer/MetadataNormal.cc:100
>> #6 0x000000000056f0e5 in Hypertable::AccessGroup::update_files_column
>> (this=0x1bc2a00)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>> RangeServer/AccessGroup.cc:571
>> #7 0x000000000056de6f in Hypertable::AccessGroup::run_compaction
>> (this=0x1bc2a00, timestamp=
>> {logical = 1222151526557033001, real = 1222151526557728000},
>> major=false)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>> RangeServer/AccessGroup.cc:422
>> #8 0x000000000055d5dc in Hypertable::Range::run_compaction
>> (this=0x19a1000, major=false)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>> RangeServer/Range.cc:681
>> #9 0x000000000055d4e9 in Hypertable::Range::compact (this=0x19a1000,
>> major=false)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>> RangeServer/Range.cc:666
>> #10 0x0000000000556ea9 in
>> Hypertable::MaintenanceTaskCompaction::execute (this=0x1036150)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>> RangeServer/MaintenanceTaskCompaction.cc:38
>> #11 0x0000000000541a4f in
>> Hypertable::MaintenanceQueue::Worker::operator() (this=0x5221c1a8)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>> RangeServer/MaintenanceQueue.h:108
>> #12 0x0000000000541885 in
>>
>> boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
>> void>::invoke (
>> [EMAIL PROTECTED])
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/
>> boost/function/function_template.hpp:158
>> #13 0x0000002a95b8adc7 in boost::function0<void,
>> std::allocator<boost::function_base> >::operator() ()
>> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.
>> 1.34.1
>> #14 0x0000002a95b8a407 in boost::thread_group::join_all () from /home/
>> ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
>> #15 0x000000302b80610a in start_thread () from /lib64/tls/
>> libpthread.so.0
>> #16 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
>> #17 0x0000000000000000 in ?? ()
>> (gdb) frame 1
>> #1 0x00000000006354d8 in Hypertable::InetAddr::initialize
>> (addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/
>> InetAddr.cc:71
>> 71 memcpy(&addr->sin_addr.s_addr, he->h_addr_list[0],
>> sizeof(uint32_t));
>> (gdb) x/x &addr->sin_addr.s_addr
>> 0x284d0f4: 0x00000000
>> (gdb) x/x he->h_addr_list[0]
>> 0xa6dc00: 0xa319410a
>>
>> The same problem happened twice today, in the same line of code, with
>> the same stack trace, but on a different server.
>>
>> Does anybody have any idea how this happens?
>>
>> Donald
>>
>>
>>
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
