Hi Pany, Can you try with the 0.9.5.0 "pre4" release? We overhauled the code specifically to fix a number of stability issues and the symptom that you describe sounds like one of the bugs that we fixed.
- Doug On Mon, May 9, 2011 at 6:59 PM, Pany Yue <[email protected]> wrote: > Hi, > > I deployed Hypertable cluster on one machine. using LocalBroker. There are > 5 client, every client has 50 threads, they do scan and insert on two table > concurrently. > each table has 100 columns. > One days after the system running, I found clients cann't insert or scan > anything in rangeserver, then I used ht shell to test scan, I just got the > namespace and table listing, > but it blocked when I used "select * from table"; > > After a litter while, all clients are cored. then after minutes, > RangeServer cored and exit; > > here are the bt for client: > (gdb) bt > #0 0x00002aab0c77c3d0 in ?? () > #1 0x00002aaeeeefc21a in Hypertable::IOHandler::deliver_event () from > /home/combo/usr/lib/libHyperComm.so > #2 0x00002aaeeeef995d in Hypertable::IOHandlerData::handle_message_body () > from /home/combo/usr/lib/libHyperComm.so > #3 0x00002aaeeeefa198 in Hypertable::IOHandlerData::handle_event () from > /home/combo/usr/lib/libHyperComm.so > #4 0x00002aaeeef10f6d in Hypertable::ReactorRunner::operator() () from > /home/combo/usr/lib/libHyperComm.so > #5 0x00002aaeeef0f900 in > boost::detail::thread_data<Hypertable::ReactorRunner>::run () from > /home/combo/usr/lib/libHyperComm.so > #6 0x00002aaeef782bbb in thread_proxy () from > /home/combo/iprocess_client/lib/libboost_thread.so.1.43.0 > #7 0x00000036a9c0673d in start_thread () from /lib64/libpthread.so.0 > #8 0x00000036a90d3f6d in clone () from /lib64/libc.so.6 > > client log: > > 1304935833 WARN Hypertable : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/AsyncComm/IOHandlerData.cc:590) > Received respon > se for non-pending event (id=4424064,version=1,total_len=42) > 1304935833 WARN Hypertable : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/AsyncComm/IOHandlerData.cc:590) > Received respon > se for non-pending event (id=4424001,version=1,total_len=42) > 1304935833 WARN Hypertable : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/AsyncComm/IOHandlerData.cc:590) > Received respon > se for non-pending event (id=4424007,version=1,total_len=42) > 1304939118 ERROR Hypertable : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/Lib/TableMutator.cc:51) > caught std: > :exception: > 1304939118 ERROR Hypertable : ~TableMutator > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/Lib/TableMutator.cc:8 > 0): Hypertable::Exception: - HYPERTABLE request timeout > at void > Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&) > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/ > src/cc/Hypertable/Lib/TableMutator.cc:405) > at bool > Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&) > (/home/itlanger/hypertable/src_for_build/hypertable-0. > 9.4.3-alpha/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71): , > final flush > > > Here are the RangeServer log: > > 1304938467 ERROR Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/AsyncComm/Comm.cc:273) > No connection for 10.20.130.101:54645 - COMM not connected > 1304938467 ERROR Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:1939) > Problem sending OK response - COMM not connected > 1304938472 INFO Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/Lib/CommitLog.cc:255) > Purging commit log fragments with latest revision older than > 1304933365168229004 > 1304938472 INFO Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/Lib/CommitLog.cc:275) > clgc LOG FRAGMENT PURGE breaking because 1304934571714547031 >= > 1304933365168229004 > 1304938472 ERROR Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/AsyncComm/Comm.cc:273) > No connection for 10.20.130.101:54645 - COMM not connected > 1304938472 ERROR Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:1939) > Problem sending OK response - COMM not connected > 1304938472 INFO Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:171) > Memory Statistics (MB): VM=8161.32, RSS=6644.96, tracked=3445.58, > computed=3445.58 limit=8032.00 > 1304938472 INFO Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:176) > Memory Allocation: BlockCache=87.06% BlockIndex=3.08% BloomFilter=0.44% > CellCache=7.96% ShadowCache=0.00% QueryCache=1.45% > 1304938477 ERROR Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/AsyncComm/Comm.cc:273) > No connection for 10.20.130.101:54645 - COMM not connected > ... > STAT 1/0[29657_ > http://www.cartech8.com/forum-29308-1.html..29766_http://www.z0760.com/zhongshan-7179-1-1.html](default)cumulative_size > 84239734 <= prune_threshold 200000000 > STAT 1/0[29766_ > http://www.z0760.com/zhongshan-7179-1-1.html..29878_http://bbs.manzuo.com/redirect.php-tid-63928-goto-lastpost](default)cumulative_size > 84239734 <= prune_threshold 200000000 > STAT 1/0[29878_ > http://bbs.manzuo.com/redirect.php-tid-63928-goto-lastpost..ÿÿ](default)cumulative_size > 84239734 <= prune_threshold 200000000 > STAT 1/1[..14993_2562e20f038e03a2ad08fe61f460c0c2](default) cumulative_size > 84239734 <= prune_threshold 200000000 > STAT 1/1[14993_2562e20f038e03a2ad08fe61f460c0c2..ÿÿ](default) > cumulative_size 84239734 <= prune_threshold 200000000 > 1304939767 INFO Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) > Memory Usage: 3460595399 bytes > 1304939778 INFO Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/TableInfo.cc:184) > Adding range 5/0[..ÿÿ] to TableInfo end row = ÿÿ > 1304939778 INFO Hypertable.RangeServer : > (/home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/RangeServer.cc:1183) > Successfully loaded range 5/0[..ÿÿ] > Hypertable.RangeServer: > /home/itlanger/hypertable/src_for_build/hypertable-0.9.4.3-alpha/src/cc/Hypertable/RangeServer/TableInfo.cc:181: > void Hypertable::TableInfo::add_range(Hypertable::RangePtr&): Assertion > `iter == m_range_map.end()' failed. > > > There are approximately 100,000 cell/s inserted, and 50,000 cells/s > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
