Hi Kenny, I think I may know what's going on here. Can you try your test again with the following changes:
1. Revert the Hypertable.RangeServer.MemoryLimit changes. In other words, leave the RangeServer memory properties at their defaults (60% of physical RAM). 2. Add the following property to your hypertable.cfg to disable the block cache: Hypertable.RangeServer.BlockCache.MaxMemory=0 I suspect that the RangeServer is fighting with the OS file cache for RAM. Disabling the block cache is not as bad as you might think. The OS will cache the compressed blocks in its file cache. - Doug On Fri, Aug 17, 2012 at 7:24 AM, Kenny F. <[email protected]> wrote: > )) > I hope this issue help you to locate the problem. > Hypertable's RangeServers use ALL memory(not all FREE memory) when have no > memory limit (or high memory limit). > It is easy to watch "Block Cache Max Memory" at RangeServer Statistics > (monitoring). > This issue appeared between Hypertable 0.9.5.0 and Hypertable 0.9.5.4. > I hope this bug will be fix. > > > On Thursday, August 16, 2012 4:19:53 PM UTC+3, Christoph Rupp wrote: > >> Well, by default the RangeServer uses 60%, but that memory is used for >> caching CellStores. >> >> If there's additional memory required to fetch data for a single request, >> then it can exceed the limit for a short while. It seems that this might >> have happened here, If you have very large scans with lots of data (and in >> addition some of the memory was allocated by apache). >> >> But i am glad that we worked it out :) >> >> bye >> Christoph >> >> 2012/8/16 Kenny F. <[email protected]> >> >> Thanks! >>> >>> I've already done it, it helped and post graphs (today) for you to >>> interpret them. >>> Earlier I used Hypertable.RangeServer.**MemoryL**imit.Percentage=20 or >>> 30 or 40 and it didn't help me at all. >>> Now I use Hypertable.RangeServer.**MemoryL**imit=1900Mb (see my post >>> today with graphs) >>> >>> I think, the reason is when we have no limit or high limit, RangeServer >>> use ALL memory (see graph "Block Cache Max Memory"), even if it is fill at >>> 95% or more. >>> >>> >>> >>> >>> On Thursday, August 16, 2012 4:04:31 PM UTC+3, Christoph Rupp wrote: >>> >>>> Hi Kenny, >>>> >>>> can you try reducing the memory usage of the RangeServer? By default >>>> the RangeServer uses 60% of the memory. >>>> >>>> There are two properties: >>>> >>>> Hypertable.RangeServer.**MemoryL**imit or Hypertable.RangeServer.** >>>> MemoryL**imit.Percentage. >>>> >>>> You can use either of them. >>>> >>>> Bye >>>> Christoph >>>> >>>> 2012/8/16 Kenny F. <[email protected]> >>>> >>>> >Is it possible that there is some other process (or set of processes) >>>>> on the machine at the time of the crash that is using up all virtual >>>>> memory? >>>>> I don't think so. >>>>> First, I have 2 memory greed processes only: Hypertable and Apache. >>>>> Second, I didn't change the environment. And restarting the system >>>>> didn't solve the issue. >>>>> Third, RangeServer didn't live more then several hours. >>>>> And I don't remember, when RangeServer crashes immediately. >>>>> >>>>> >>>>> On Thursday, August 16, 2012 3:02:39 PM UTC+3, Doug Judd wrote: >>>>> >>>>>> Hi Kenny, >>>>>> >>>>>> One other thought. Is it possible that there is some other process >>>>>> (or set of processes) on the machine at the time of the crash that is >>>>>> using >>>>>> up all virtual memory? Try running 'free' at regular intervals during >>>>>> your >>>>>> test to see if the system is running out of swap space. >>>>>> >>>>>> - Doug >>>>>> >>>>>> On Tue, Aug 14, 2012 at 6:51 AM, Kenny F. <[email protected]> wrote: >>>>>> >>>>>>> 0.9.6.0.95b0abc tracks: >>>>>>> >>>>>>> # screen -r >>>>>>> >>>>>>> >>>>>>> Program received signal SIGABRT, Aborted. >>>>>>> [Switching to Thread 0x99ec9b70 (LWP 15640)] >>>>>>> 0xffffe424 in __kernel_vsyscall () >>>>>>> >>>>>>> >>>>>>> # where >>>>>>> >>>>>>> #0 0xffffe424 in __kernel_vsyscall () >>>>>>> #1 0xb7b28781 in raise () from /lib/i686/cmov/libc.so.6 >>>>>>> #2 0xb7b2bbb2 in abort () from /lib/i686/cmov/libc.so.6 >>>>>>> #3 0xb7d34959 in __gnu_cxx::__verbose_**terminate****_handler() () >>>>>>> from /opt/hypertable/0.9.6.0.**95b0ab****c/lib/libstdc++.so.6 >>>>>>> #4 0xb7d32865 in ?? () from /opt/hypertable/0.9.6.0.**95b0ab**** >>>>>>> c/lib/libstdc++.so.6 >>>>>>> #5 0xb7d328a2 in std::terminate() () from /opt/hypertable/0.9.6.0.* >>>>>>> *95b0ab****c/lib/libstdc++.so.6 >>>>>>> #6 0xb7d329da in __cxa_throw () from /opt/hypertable/0.9.6.0.** >>>>>>> 95b0ab****c/lib/libstdc++.so.6 >>>>>>> *#7 0xb7d33033 in operator new(unsigned int) () from >>>>>>> /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6 >>>>>>> #8 0xb7d3311d in operator new[](unsigned int) () from >>>>>>> /opt/hypertable/0.9.6.0.95b0abc/lib/libstdc++.so.6* >>>>>>> #9 0x0869ef81 in Hypertable::DynamicBuffer::**gro****w >>>>>>> (this=0x99ec7ff4, new_size=1000004, nocopy=false) at >>>>>>> /root/src/hypertable/src/cc/**Co****mmon/DynamicBuffer.h:120 >>>>>>> #10 0x0869f095 in Hypertable::DynamicBuffer::**res****erve >>>>>>> (this=0x99ec7ff4, len=1000004, nocopy=false) at >>>>>>> /root/src/hypertable/src/cc/ >>>>>>> **Co****mmon/DynamicBuffer.h:72 >>>>>>> #11 0x08773887 in Hypertable::FillScanBlock (scanner=..., dbuf=..., >>>>>>> buffer_size=1000000) at /root/src/hypertable/src/cc/**Hy**** >>>>>>> pertable/RangeServer/**FillScanB****lock.cc:104 >>>>>>> #12 0x086628c1 in Hypertable::RangeServer::**creat****e_scanner >>>>>>> (this=0x8c95460, cb=0x99ec92a8, table=0x99ec929c, range_spec=0x99ec9290, >>>>>>> scan_spec=0x99ec9200, cache_key=0x99ec9278) >>>>>>> at /root/src/hypertable/src/cc/**Hy****pertable/RangeServer/** >>>>>>> RangeServ****er.cc:1371 >>>>>>> #13 0x087d53e6 in Hypertable::**RequestHandlerCrea****teScanner::**run >>>>>>> (this=0x331d9840) at /root/src/hypertable/src/cc/**Hy**** >>>>>>> pertable/RangeServer/**RequestHa****ndlerCreateScanner.**cc:59 >>>>>>> #14 0x0862debc in Hypertable::ApplicationQueue::******Worker::operator() >>>>>>> (this=0x8c83cf8) at /root/src/hypertable/src/cc/**As**** >>>>>>> yncComm/ApplicationQueue.h:**172 >>>>>>> #15 0x0862df18 in boost::detail::thread_data<**Hyp**** >>>>>>> ertable::ApplicationQueue::**Wor****ker>::run (this=0x8c83c28) at >>>>>>> /usr/local/include/boost/**threa****d/detail/thread.hpp:61 >>>>>>> #16 0xb7ebfe68 in thread_proxy () from /opt/hypertable/0.9.6.0.** >>>>>>> 95b0ab****c/lib/libboost_thread.**so.1.44.****0 >>>>>>> #17 0xb7e05955 in start_thread () from /lib/i686/cmov/libpthread.so.0 >>>>>>> #18 0xb7bca5ee in clone () from /lib/i686/cmov/libc.so.6 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Hypertable Development" group. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/**ms****g/hypertable-dev/-/**mV9fO9SeT_* >>>>>>> *I**J<https://groups.google.com/d/msg/hypertable-dev/-/mV9fO9SeT_IJ> >>>>>>> . >>>>>>> >>>>>>> To post to this group, send email to hyperta...@googlegroups.**com. >>>>>>> To unsubscribe from this group, send email to hypertable-de...@** >>>>>>> googlegroups.****com. >>>>>>> >>>>>>> For more options, visit this group at http://groups.google.com/** >>>>>>> group****/hypertable-dev?hl=en<http://groups.google.com/group/hypertable-dev?hl=en> >>>>>>> . >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Doug Judd >>>>>> CEO, Hypertable Inc. >>>>>> >>>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Hypertable Development" group. >>> To view this discussion on the web visit https://groups.google.com/d/** >>> msg/hypertable-dev/-/**CWqXo6VjsGQJ<https://groups.google.com/d/msg/hypertable-dev/-/CWqXo6VjsGQJ> >>> . >>> To post to this group, send email to hyperta...@googlegroups.**com. >>> To unsubscribe from this group, send email to hypertable-de...@** >>> googlegroups.com. >>> >>> For more options, visit this group at http://groups.google.com/** >>> group/hypertable-dev?hl=en<http://groups.google.com/group/hypertable-dev?hl=en> >>> . >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/hypertable-dev/-/WDrfBDUZHZMJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- Doug Judd CEO, Hypertable Inc. -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
