I believe I've found the problem: In function InetAddr::initialize(),
the non-reentrant function gethostbyname() is used unprotected. I've
replaced it with gethostbyname_r() and tested 3 more times, the
problem didn't occur any more.
According to its manpage, gethostbyname_r() seems to be only available
in Linux. So a simple substitution may influence Hypertable's
portability. I wonder if there is any other more portable solution?
Donald
On Tue, Sep 23, 2008 at 11:24 PM, Liu Kejia(Donald)
<[EMAIL PROTECTED]> wrote:
> Hi Doug,
>
> I know this command, you've taught me many times :)
>
> I did examine other threads, but find nothing unusual. Full story followed:
>
> $ gdb -c core.333 hypertable/0.9.0.10/bin/Hypertable.RangeServer
> [...]
> Core was generated by
> `/home/ht4/hypertable/0.9.0.10/bin/Hypertable.RangeServer
> --pidfile=/home/ht4/hy'.
> Program terminated with signal 11, Segmentation fault.
> [...]
> #0 0x000000302af71900 in memcpy () from /lib64/tls/libc.so.6
> (gdb) thread apply all where
>
> Thread 30 (process 333):
> #0 0x000000302b806ffb in pthread_join () from /lib64/tls/libpthread.so.0
> #1 0x0000002a95b8a1be in boost::thread::join () from
> /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #2 0x0000002a95b8a336 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #3 0x000000000051b5f8 in Hypertable::ApplicationQueue::join (this=0xa6bdc0)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:189
> #4 0x000000000051ae22 in main (argc=3, argv=0x7fbffffba8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/main.cc:171
>
> Thread 29 (process 341):
> #0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
> #1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
> (this=0x40a001a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
> #2 0x000000000062c1e3 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
> void>::invoke (
> [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #3 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #4 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #7 0x0000000000000000 in ?? ()
>
> Thread 28 (process 342):
> #0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
> #1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
> (this=0x414011a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
> #2 0x000000000062c1e3 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
> void>::invoke (
> [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #3 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
>
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #4 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #7 0x0000000000000000 in ?? ()
>
> Thread 27 (process 343):
> #0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
> #1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
> (this=0x41e021a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
> #2 0x000000000062c1e3 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
> void>::invoke (
> [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #3 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #4 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #7 0x0000000000000000 in ?? ()
>
> Thread 26 (process 344):
> #0 0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
> #1 0x000000000062dbf4 in Hypertable::ReactorRunner::operator()
> (this=0x428031a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ReactorRunner.cc:61
> #2 0x000000000062c1e3 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ReactorRunner,
> void>::invoke (
> [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #3 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #4 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #5 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #6 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #7 0x0000000000000000 in ?? ()
>
> Thread 25 (process 345):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6af30, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6af30,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x0000000000622285 in Hypertable::ConnectionManager::operator()
> (this=0x432041a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ConnectionManager.cc:299
> #4 0x000000000051f443 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ConnectionManager,
> void>::invoke (
> [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 24 (process 346):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x43c051a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 23 (process 347):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x446061a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 22 (process 348):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x450071a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 21 (process 349):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x45a081a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 20 (process 350):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x464091a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 19 (process 351):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x46e0a1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 18 (process 352):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4780b1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 17 (process 353):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4820c1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 16 (process 354):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x48c0d1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 15 (process 355):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4960e1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 14 (process 356):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4a00f1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 13 (process 357):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4aa101a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 12 (process 358):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4b4111a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 11 (process 359):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4be121a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 10 (process 360):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4c8131a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 9 (process 361):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4d2141a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 8 (process 362):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4dc151a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 7 (process 363):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4e6161a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 6 (process 364):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4f0171a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 5 (process 365):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0xa6be80, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0xa6be80,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x000000000051e4c0 in
> Hypertable::ApplicationQueue::Worker::operator() (this=0x4fa181a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/ApplicationQueue.h:96
> #4 0x000000000051e441 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::ApplicationQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #5 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #6 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #7 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> Thread 4 (process 366):
> #0 0x000000302b8089aa in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x000000000051eb00 in boost::condition::do_wait<boost::mutex>
> (this=0x50418008, [EMAIL PROTECTED])
> at /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:150
> #2 0x000000000051eacb in
> boost::condition::wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (this=0x50418008,
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:93
> #3 0x00000000006175ac in
> Hypertable::DispatchHandlerSynchronizer::wait_for_reply
> (this=0x50417f80, [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/AsyncComm/DispatchHandlerSynchronizer.cc:59
> #4 0x000000000058f20d in Hypertable::DfsBroker::Client::close
> (this=0xa6fe10, fd=82320)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/DfsBroker/Lib/Client.cc:224
> #5 0x000000000057b3b9 in ~CellStoreV0 (this=0x3829c00)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/CellStoreV0.cc:66
> #6 0x000000000051c8e1 in Hypertable::intrusive_ptr_release (rc=0x3829c00)
> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/ReferenceCount.h:73
> #7 0x000000000055f57a in ~intrusive_ptr (this=0x113f5908) at
> /usr/local/include/boost-1_34_1/boost/intrusive_ptr.hpp:83
> #8 0x000000000058ac6f in ~CellStoreScannerV0 (this=0x113f58f0)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/CellStoreScannerV0.cc:232
> #9 0x00000000005814f3 in ~MergeScanner (this=0x2ac9208000)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MergeScanner.cc:53
> #10 0x000000000051c8e1 in Hypertable::intrusive_ptr_release (rc=0x2ac9208000)
> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/ReferenceCount.h:73
> #11 0x0000000000524e34 in ~intrusive_ptr (this=0x50418ef0) at
> /usr/local/include/boost-1_34_1/boost/intrusive_ptr.hpp:83
> #12 0x000000000056e00c in Hypertable::AccessGroup::run_compaction
> (this=0xa7a7c00, timestamp=
> {logical = 1222151526556990005, real = 1222151526557728000}, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:430
> #13 0x000000000055d5dc in Hypertable::Range::run_compaction
> (this=0x2bda9900, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
> #14 0x000000000055d4e9 in Hypertable::Range::compact (this=0x2bda9900,
> major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
> #15 0x0000000000556ea9 in
> Hypertable::MaintenanceTaskCompaction::execute (this=0x2ab7fe88c0)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
> #16 0x0000000000541a4f in
> Hypertable::MaintenanceQueue::Worker::operator() (this=0x504191a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
> #17 0x0000000000541885 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #18 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #19 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #20 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #21 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #22 0x0000000000000000 in ?? ()
>
> Thread 3 (process 367):
> #0 0x000000302b808b9f in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/tls/libpthread.so.0
> #1 0x0000002a95b8886c in boost::detail::condition_impl::do_timed_wait ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #2 0x0000000000541dee in
> boost::condition::do_timed_wait<boost::mutex> (this=0x2aeabf3e48,
> [EMAIL PROTECTED],
> [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:178
> #3 0x0000000000541da9 in
> boost::condition::timed_wait<boost::detail::thread::scoped_lock<boost::mutex>
>> (
> this=0x2aeabf3e48, [EMAIL PROTECTED], [EMAIL PROTECTED]) at
> /usr/local/include/boost-1_34_1/boost/thread/condition.hpp:112
> #4 0x00000000005d04e2 in
> Hypertable::TableMutatorCompletionCounter::wait_for_completion
> (this=0x2aeabf3e20,
> [EMAIL PROTECTED]) at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:64
> #5 0x00000000005cf2b8 in
> Hypertable::TableMutatorScatterBuffer::wait_for_completion
> (this=0x2aeabf3d80, [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:255
> #6 0x00000000005ccaef in
> Hypertable::TableMutator::wait_for_previous_buffer (this=0x2af826e0c0,
> [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutator.cc:253
> #7 0x00000000005cc917 in Hypertable::TableMutator::flush (this=0x2af826e0c0)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutator.cc:206
> #8 0x0000000000585708 in Hypertable::MetadataNormal::write_files
> (this=0x50e19c50, [EMAIL PROTECTED],
> [EMAIL PROTECTED]) at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MetadataNormal.cc:101
> #9 0x000000000056f0e5 in Hypertable::AccessGroup::update_files_column
> (this=0x2ac64ca800)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:571
> #10 0x000000000056de6f in Hypertable::AccessGroup::run_compaction
> (this=0x2ac64ca800, timestamp=
> {logical = 1222151526557045002, real = 1222151526557728000}, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:422
> #11 0x000000000055d5dc in Hypertable::Range::run_compaction
> (this=0x9868480, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
> #12 0x000000000055d4e9 in Hypertable::Range::compact (this=0x9868480,
> major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
> #13 0x0000000000556ea9 in
> Hypertable::MaintenanceTaskCompaction::execute (this=0x3bf883f0)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
> #14 0x0000000000541a4f in
> Hypertable::MaintenanceQueue::Worker::operator() (this=0x50e1a1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
> #15 0x0000000000541885 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #16 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #17 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #18 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #19 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #20 0x0000000000000000 in ?? ()
>
> Thread 2 (process 368):
> #0 0x00000000005f8132 in _lzo1x_1_do_compress (in=0x2ae664c000
> "\034e9d31a63d844791c", in_len=264505, out=0x2afb56701a "",
> out_len=0x5181a568, wrkmem=0x2ac4607000)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/lzo/minilzo.c:2713
> #1 0x00000000005f82fb in lzo1x_1_compress (in=0x2ae664c000
> "\034e9d31a63d844791c", in_len=264505, out=0x2afb56701a "",
> out_len=0x5181a568, wrkmem=0x2ac4607000)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/lzo/minilzo.c:2785
> #2 0x00000000005dc9f6 in
> Hypertable::BlockCompressionCodecLzo::deflate (this=0x29463840,
> [EMAIL PROTECTED],
> [EMAIL PROTECTED], [EMAIL PROTECTED], reserve=0)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/BlockCompressionCodecLzo.cc:74
> #3 0x000000000057c61d in Hypertable::CellStoreV0::finalize
> (this=0xa426e00, [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/CellStoreV0.cc:211
> #4 0x000000000056d70c in Hypertable::AccessGroup::run_compaction
> (this=0xef02400, timestamp=
> {logical = 1222151526557057001, real = 1222151526557728000}, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:361
> #5 0x000000000055d5dc in Hypertable::Range::run_compaction
> (this=0x1a8ab900, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
> #6 0x000000000055d4e9 in Hypertable::Range::compact (this=0x1a8ab900,
> major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
> #7 0x0000000000556ea9 in
> Hypertable::MaintenanceTaskCompaction::execute (this=0x171a6ab8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
> #8 0x0000000000541a4f in
> Hypertable::MaintenanceQueue::Worker::operator() (this=0x5181b1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
> #9 0x0000000000541885 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #10 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #11 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #12 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #13 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #14 0x0000000000000000 in ?? ()
> Current language: auto; currently c
>
> Thread 1 (process 369):
> #0 0x000000302af71900 in memcpy () from /lib64/tls/libc.so.6
> #1 0x00000000006354d8 in Hypertable::InetAddr::initialize
> (addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/InetAddr.cc:71
> #2 0x00000000005e855a in Hypertable::LocationCache::location_to_addr
> (location=0x1311fd8 "10.65.25.163_38060",
> [EMAIL PROTECTED]) at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/LocationCache.cc:256
> #3 0x00000000005ce19c in Hypertable::TableMutatorScatterBuffer::set
> (this=0xff4c900, [EMAIL PROTECTED], value=0x165e5c18,
> value_len=639, [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:72
> #4 0x00000000005cc05b in Hypertable::TableMutator::set
> (this=0x2ab21ec240, timestamp=0, [EMAIL PROTECTED], value=0x165e5c18,
> value_len=639) at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/TableMutator.cc:93
> #5 0x00000000005856f7 in Hypertable::MetadataNormal::write_files
> (this=0x5221bc50, [EMAIL PROTECTED], [EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MetadataNormal.cc:100
> #6 0x000000000056f0e5 in Hypertable::AccessGroup::update_files_column
> (this=0x1bc2a00)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:571
> #7 0x000000000056de6f in Hypertable::AccessGroup::run_compaction
> (this=0x1bc2a00, timestamp=
> {logical = 1222151526557033001, real = 1222151526557728000}, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/AccessGroup.cc:422
> #8 0x000000000055d5dc in Hypertable::Range::run_compaction
> (this=0x19a1000, major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:681
> #9 0x000000000055d4e9 in Hypertable::Range::compact (this=0x19a1000,
> major=false)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/Range.cc:666
> #10 0x0000000000556ea9 in
> Hypertable::MaintenanceTaskCompaction::execute (this=0x1036150)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceTaskCompaction.cc:38
> #11 0x0000000000541a4f in
> Hypertable::MaintenanceQueue::Worker::operator() (this=0x5221c1a8)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
> #12 0x0000000000541885 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #13 0x0000002a95b8adc7 in boost::function0<void,
> std::allocator<boost::function_base> >::operator() ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #14 0x0000002a95b8a407 in boost::thread_group::join_all ()
> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
> #15 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
> #16 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
> #17 0x0000000000000000 in ?? ()
> (gdb) thread 2
> [Switching to thread 2 (process 368)]#0 0x00000000005f8132 in
> _lzo1x_1_do_compress (
> in=0x2ae664c000 "\034e9d31a63d844791c", in_len=264505,
> out=0x2afb56701a "", out_len=0x5181a568, wrkmem=0x2ac4607000)
> at
> /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/lzo/minilzo.c:2713
> 2713 while (ip < end && *m == *ip)
> (gdb) p/x ip
> $8 = 0x2ae665bcab
> (gdb) p/x end
> $9 = 0x2ae668c939
> (gdb) p *ip
> $10 = 9 '\t'
> (gdb) p *m
> $11 = 9 '\t'
>
> Donald
>
>
> On Tue, Sep 23, 2008 at 10:55 PM, Doug Judd <[EMAIL PROTECTED]> wrote:
>> Hi Donald,
>>
>> The segfault may have happened in a different thread. To view the stack
>> traces for all threads, issue this command:
>>
>> (gdb) thread apply all where
>>
>> You can then switch threads with the 'thread <tid>' command. Try examining
>> the other threads to see if they were the cause.
>>
>> - Doug
>>
>> On Tue, Sep 23, 2008 at 6:01 AM, donald <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi All,
>>>
>>> I meet this strange problem while loading 500GB log data into 7
>>> Hypertable Range Servers, one range server dies with a core dump.
>>> According to gdb output, the core is generated due to segmentation
>>> fault in memcpy(). However, I can access both the source and
>>> destination address in gdb:
>>>
>>> $ gdb -c core.333 Hypertable.RangeServer
>>> [...]
>>> (gdb) where
>>> #0 0x000000302af71900 in memcpy () from /lib64/tls/libc.so.6
>>> #1 0x00000000006354d8 in Hypertable::InetAddr::initialize
>>> (addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/
>>> InetAddr.cc:71
>>> #2 0x00000000005e855a in Hypertable::LocationCache::location_to_addr
>>> (location=0x1311fd8 "10.65.25.163_38060", [EMAIL PROTECTED])
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/
>>> LocationCache.cc:256
>>> #3 0x00000000005ce19c in Hypertable::TableMutatorScatterBuffer::set
>>> (this=0xff4c900, [EMAIL PROTECTED], value=0x165e5c18,
>>> value_len=639, [EMAIL PROTECTED])
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/
>>> TableMutatorScatterBuffer.cc:72
>>> #4 0x00000000005cc05b in Hypertable::TableMutator::set
>>> (this=0x2ab21ec240, timestamp=0, [EMAIL PROTECTED], value=0x165e5c18,
>>> value_len=639) at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/
>>> Hypertable/Lib/TableMutator.cc:93
>>> #5 0x00000000005856f7 in Hypertable::MetadataNormal::write_files
>>> (this=0x5221bc50, [EMAIL PROTECTED], [EMAIL PROTECTED])
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>>> RangeServer/MetadataNormal.cc:100
>>> #6 0x000000000056f0e5 in Hypertable::AccessGroup::update_files_column
>>> (this=0x1bc2a00)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>>> RangeServer/AccessGroup.cc:571
>>> #7 0x000000000056de6f in Hypertable::AccessGroup::run_compaction
>>> (this=0x1bc2a00, timestamp=
>>> {logical = 1222151526557033001, real = 1222151526557728000},
>>> major=false)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>>> RangeServer/AccessGroup.cc:422
>>> #8 0x000000000055d5dc in Hypertable::Range::run_compaction
>>> (this=0x19a1000, major=false)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>>> RangeServer/Range.cc:681
>>> #9 0x000000000055d4e9 in Hypertable::Range::compact (this=0x19a1000,
>>> major=false)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>>> RangeServer/Range.cc:666
>>> #10 0x0000000000556ea9 in
>>> Hypertable::MaintenanceTaskCompaction::execute (this=0x1036150)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>>> RangeServer/MaintenanceTaskCompaction.cc:38
>>> #11 0x0000000000541a4f in
>>> Hypertable::MaintenanceQueue::Worker::operator() (this=0x5221c1a8)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
>>> RangeServer/MaintenanceQueue.h:108
>>> #12 0x0000000000541885 in
>>>
>>> boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
>>> void>::invoke (
>>> [EMAIL PROTECTED])
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/
>>> boost/function/function_template.hpp:158
>>> #13 0x0000002a95b8adc7 in boost::function0<void,
>>> std::allocator<boost::function_base> >::operator() ()
>>> from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.
>>> 1.34.1
>>> #14 0x0000002a95b8a407 in boost::thread_group::join_all () from /home/
>>> ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
>>> #15 0x000000302b80610a in start_thread () from /lib64/tls/
>>> libpthread.so.0
>>> #16 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
>>> #17 0x0000000000000000 in ?? ()
>>> (gdb) frame 1
>>> #1 0x00000000006354d8 in Hypertable::InetAddr::initialize
>>> (addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
>>> at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/
>>> InetAddr.cc:71
>>> 71 memcpy(&addr->sin_addr.s_addr, he->h_addr_list[0],
>>> sizeof(uint32_t));
>>> (gdb) x/x &addr->sin_addr.s_addr
>>> 0x284d0f4: 0x00000000
>>> (gdb) x/x he->h_addr_list[0]
>>> 0xa6dc00: 0xa319410a
>>>
>>> The same problem happened twice today, in the same line of code, with
>>> the same stack trace, but on a different server.
>>>
>>> Does anybody have any idea how this happens?
>>>
>>> Donald
>>>
>>>
>>>
>>
>>
>> >>
>>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
