Thanks for sharing this.... Can you please switch off the cluster, then only start dfsbrokers and hyperspace, and run this:
/opt/hypertable/current/bin/ht metalog_dump /hypertable/servers/rs1/log/rsml and grep the output for "load_acknowledged=false". If that flag is set then the ranges are not compacted, because the master is waiting for the rangeserver to confirm that it has loaded the range. Thanks Christoph 2012/6/20 BigQiao <[email protected]> > I think I figure it out. > The memory is cost by Ranges which load by replay_update() in > RangeServer.cc > those Ranges will not be compact because their needs_compaction=false; > > So I add one line to RangeServer.cc:3616, RangeServer::replay_load_range > > 3612 schema = table_info->get_schema(); > 3613 > 3614 range = new Range(m_master_client, schema, range_entity, > live_table_info.get()); > 3615 //## > 3616 range->set_needs_compaction(true); //## Qiao Fix Memory Bug > 2012-06-20 > 3617 //## > 3618 range->recovery_initialize(); > 3619 > 3620 table_info->add_range(range); > > After replay finish, it will start to compact cellcaches. > > > > We have set Hypertable.RangeServer.**MemoryLimit=4GB, but rangeserver >> still run out of memory(almost cost 10GB), and 99% cost by CellCache, >> >> I wait it for almost 1 hour, but nothing changed. I belive the >> compactions stopped. >> >> >> Hi, >>> >>> "Application queue PAUSED due to low memory condition" is actually quite >>> common and usually not a problem, unless you have MANY of them. These are >>> usually caused either by low memory (because load is too high) or because >>> hypertable fails to run compactions. We had a bug in 0.9.5.4 which did not >>> reset some internal flags correctly, thus stopping compactions, and the >>> cellcaches were not flushed to disk. but this was fixed in 0.9.5.6 (or >>> 0.9.5.5, if i remember correctly). >>> >>> There are a couple of options you can use to control memory behavior: >>> >>> >>> --Hypertable.RangeServer.**MemoryLimit arg >>> ** RangeServer memory limit >>> --Hypertable.RangeServer.**MemoryLimit.Percentage arg (=60) >>> ** RangeServer memory limit >>> specified as >>> ** percentage of physical RAM >>> --Hypertable.RangeServer.**LowMemoryLimit.Percentage arg (=10) >>> ** Amount of memory to free in >>> low memory >>> ** condition as percentage of >>> RangeServer >>> ** memory limit >>> --Hypertable.RangeServer.**MemoryLimit.EnsureUnused arg >>> ** Amount of unused physical >>> memory >>> --Hypertable.RangeServer.**MemoryLimit.EnsureUnused.**Percentage arg >>> ** Amount of unused physical >>> memory >>> ** specified as percentage of >>> physical RAM >>> >>> bye >>> Christoph >>> 2012/6/19 gcc.lua <[email protected]> >>> >>>> Hi,all >>>> I have use c++ stream pipe to implement map/reduce logic handle >>>> big data(hypertable/hadoop) >>>> I had search the old issues like my problem, it's like CellCache >>>> memmory fragment, >>>> and fixed use CellCachePool import in 0.9.1.0/0.9.1.1, It be removed >>>> in latest version 0.9.5.6(or sth. old version), >>>> where can I commit the Hypertable.RangeServer.log, thanks >>>> >>>> server:(16GB Memory) >>>> >>>> 20674 line: >>>> 1340084923 INFO Hypertable.RangeServer : (/home/hadoop/temp/ >>>> hypertable-0615/src/cc/**Hypertable/RangeServe r/ >>>> MaintenanceScheduler.cc:255) Memory Allocation: BlockCache=0.00% >>>> BlockIndex=0.00% BloomFilter=0.00% CellCache=99.51% ShadowCache=0.00% >>>> QueryCache=0.49% >>>> >>>> >>>> 221688 line: >>>> 1340093490 INFO Hypertable.RangeServer : (/home/hadoop/temp/ >>>> hypertable-0615/src/cc/**Hypertable/RangeServe r/RangeServer.cc: >>>> 3710) Application queue PAUSED due to low memory condition >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Hypertable Development" group. >>>> To post to this group, send email to >>>> hypertable-dev@googlegroups.**com<[email protected]> >>>> . >>>> To unsubscribe from this group, send email to >>>> hypertable-dev+unsubscribe@**googlegroups.com<hypertable-dev%[email protected]> >>>> . >>>> For more options, visit this group at http://groups.google.com/** >>>> group/hypertable-dev?hl=en<http://groups.google.com/group/hypertable-dev?hl=en> >>>> . >>>> >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/hypertable-dev/-/oNBjwrL8XIYJ. > > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
