Hi Sanjit, This config setting helped us tremendously, thank you *very* much for posting it! We were having serious problems with bloating RangeServers which would use most or all of the 12G of ram per machine and once we restarted everything with your suggested config we saw an order of magnitude of decreased memory usage for a similar workload and time period. The RangeServers now typically use around 1.1-1.3G for normal operation after 14 hours or so (eg, inserting at most one or two thousand ~400+~100 byte average cells+keys per second per RangeServer.)
Another potentially useful config setting for anyone using KFS is Kfs.Broker.Flush = true We've found that without this setting the kosmosBroker memory usage appeared to grow more quickly, perhaps around twice as fast as it does now. I've taken to shutting down Hypertable and killing all the binaries when I see memory usage start to creep up beyond something like 2G per RangeServer and 1-2G per kosmosBroker. This results in something like one restart per day which makes for an easy-to-schedule daily maintenance window for the database. I've finally figured out that a safe sequence for shutdown is: 1) Kill all ThriftBrokers and any other clients (eg, load data infile's via ./bin/hypertable) which might be interacting with the RangeServers (especially inserts) 1a) double check that the RangeServers are idle / not busy finishing anything heavyweight like compactions or what have you 2) Run "shutdown;" in the hypertable shell and verify that all of the RangeServers go down 3) Run ./bin/stop-servers.sh on the master and verify that Hypertable.Master and Hyperspace.Master are down 4) Kill all other running binaries, probably just the DFS Brokers (in our case it's kosmosBroker) So far this has allowed me to restart Hypertable without any changes to the existing data or to the performance of inserting new data. However, I wanted to get feedback on whether or not what I'm doing could get me in trouble with potential edge cases which I've missed or with upcoming changes to the architecture hypertable-dev might have planned. Cheers! Josh On Thu, Apr 2, 2009 at 1:51 PM, Sanjit Jhala <[email protected]> wrote: > In case you are running Hypertable v0.9.2.2/v0.9.2.3 without installing > tcmalloc, it is strongly recommended that you add the following parameter to > your configuration: > Hypertable.RangeServer.AccessGroup.CellCache.PageSize=512K > We have observed that on some systems Hypertable is consuming way more > memory than it needs to owing to a glibc malloc heap fragmentation issue > described here: > http://lwn.net/Articles/249113/ > http://blogs.gnome.org/alexl/2009/02/17/eternal-vigilance/ > The memory usage goes down drastically with the PageSize parameter set to > 512K instead of the current default 100K. In future releases the default > will be set to 512K. In case your Hypertable installation uses tcmalloc (as > recommended in the Hypertable install instructions) this should not be a > problem and consequently you don't need to change your configuration. > -Sanjit > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
