Hi Pan, [I'm CC'ing the list so others can benefit as well.] When free() is called, the memory is given back to the memory allocator (which may or may not give it back to the system right away). In the case of Hypertable, we link against tcmalloc <http://goog-perftools.sourceforge.net/doc/tcmalloc.html>. tcmalloc does not give the memory immediately back to the system. To verify this, I wrote a simple test program which I've attached. If you compile with tcmalloc, run it and wait for it enter poll(), you'll see that the RSS does not decrease:
$ g++ -o foo foo.cc -ltcmalloc $ ./foo 5 $ ps auxww | fgrep foo USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND doug 24076 54.4 32.0 5275268 5259260 pts/0 S+ 21:18 0:08 ./foo 5 I tried this with glibc malloc and apparently it does give memory back to the system after it is freed: $ g++ -o foo foo.cc $ ./foo 5 $ ps auxww | fgrep foo USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND doug 24292 64.6 0.0 11380 888 pts/0 S 21:37 0:11 ./foo 5 So it it looks like this behavior is specific to tcmalloc. I tried to verify if the RSS would decrease if I ran another instance of foo while the other still existed, but there wasn't enough swap configured on the system, so the second time I ran it it dumped core with "bad alloc". However, I believe if the system did have enough swap space configured, it would swap out the first instance of foo to make room for the second instance thereby reducing the RSS of the first instance. - Doug On Fri, Apr 15, 2011 at 7:10 PM, pan yue <[email protected]> wrote: > Does rangerserver alloc and free memory in it's own memory pool ? > I see PageArena.h: > 37 struct DefaultPageAllocator > { > > 38 void *allocate(size_t sz) { return std::malloc(sz); > } > 39 void deallocate(void *p) { std::free(p); > } > > 40 void freed(size_t sz) { /* mostly for effcient bulk stat reporting */ > } > 41 }; > PageArena will free the memory to Linux system, don't manager the unused > memory by itself, according to this implementation, the RSS will drop to a > low-level threshold when no insert or scan activities. > > > 2011/4/16 Doug Judd <[email protected]> > >> This is not a problem. You've given the RangeServer 60% of memory which >> comes out to about 10GB. When the RangeServer frees memory it merely gives >> it back to the heap. If there is nothing going on on the system, the RSS >> won't change. If you run a different program that actively used a large >> amount of memory, only then would you see the RSS for the RangeServer >> shrink. >> >> - Doug >> >> >> On Fri, Apr 15, 2011 at 10:13 AM, pan yue <[email protected]> wrote: >> >>> I use the second post: >>> >>> Hypertable.RangeServer.MemoryLimit.Percentage=60 >>> >>> The problem is RSS is always 8GB when no insert or scan happend, >>> According to log file: >>> >>> Memory Statistics (MB): VM=10259.90, RSS=8223.74, tracked=216.58, >>> computed=216.58 limit=8424.00 >>> >>> RangeServer only tracked 216MB. >>> >>> >>> >>> 2011/4/16 Doug Judd <[email protected]> >>> >>>> What is the config file that you're using? In your first post, you >>>> indicated that the config file contained the following lines: >>>> >>>> Hypertable.RangeServer.MemoryLimit=4096000000 >>>> Hypertable.RangeServer.MemoryLimit.Percentage=25 >>>> >>>> But in your second post, you showed a config file that contained: >>>> >>>> Hypertable.RangeServer.MemoryLimit.Percentage=60 >>>> >>>> If the RSS hits 8GB with a 25% memory limit, then that would indicate >>>> heap fragmentation. An RSS of 8GB with a 60% memory limit is normal and >>>> expected. >>>> >>>> - Doug >>>> >>>> P.S. Your client program looks ok to me. >>>> >>>> On Fri, Apr 15, 2011 at 7:19 AM, pan yue <[email protected]>wrote: >>>> >>>>> What I have seen is The RSS always 8GB for at least 5 hours after the >>>>> insert activity. >>>>> and it always 8GB for at lease 24 hours in another machine for having >>>>> the same problem which I tested yesterday. >>>>> >>>>> I don't know why, and How can I reduce heap fragmentation? >>>>> Did the HypertableClient I write have any problem? >>>>> >>>>> >>>>> >>>>> 2011/4/15 Doug Judd <[email protected]> >>>>> >>>>>> What you're seeing is heap fragmentation. During the heavy insert >>>>>> activity, the Memory usage grew up to the configured limit >>>>>> (Hypertable.RangeServer.MemoryLimit.Percentage=60). >>>>>> You can verify this by running the following command: >>>>>> >>>>>> $ fgrep "Memory Usage" Hypertable.RangeServer.log >>>>>> [...] >>>>>> 1302844863 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) >>>>>> Memory Usage: 8519348352 bytes >>>>>> 1302844864 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) >>>>>> Memory Usage: 483976199 bytes >>>>>> 1302844864 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) >>>>>> Memory Usage: 534307847 bytes >>>>>> [...] >>>>>> >>>>>> As you can see by the output, at Thu Apr 14 22:21:03 2011, the memory >>>>>> usage hit 8519348352 bytes. >>>>>> >>>>>> Once the insert activity stopped, Hypertable started to aggressively >>>>>> compact in-memory CellCaches to on-disk CellStores in order to prune back >>>>>> the commit log (for faster recovery). Once a CellCache gets compacted, >>>>>> the >>>>>> memory gets freed and subtracted from the internal memory counter. In >>>>>> the 0.9.4.3 release, the minimum prune threshold for the commit log is >>>>>> 200M >>>>>> which is why the tracked memory number drops to ~200M. When memory gets >>>>>> freed, it is not immediately reflected in the RSS. If you were to run >>>>>> some >>>>>> other program after your Hypertable test, that used a lot of memory, I >>>>>> suspect you would see the Hypertable RSS shrink towards 200MB. >>>>>> >>>>>> - Doug >>>>>> >>>>>> On Thu, Apr 14, 2011 at 10:59 PM, pan yue <[email protected]>wrote: >>>>>> >>>>>>> Thanks for Doug Judd 's reply. the following is what done when >>>>>>> RangerServer leak memory. >>>>>>> >>>>>>> 1. Deployment: I deployed hypertable-0.9.34 on one machine(16GB >>>>>>> memory, 8 cores). >>>>>>> ./start-dfsbroker.sh hadoop >>>>>>> ./start-hyperspace.sh >>>>>>> ./start-master.sh >>>>>>> ./start-rangerserver.sh >>>>>>> >>>>>>> >>>>>>> 2. test insert data: one c++ hypertable client, single thread, here >>>>>>> is the test result: >>>>>>> ./pressClient 0 1 1000000 > output.txt >>>>>>> >>>>>>> TotalBytes: 65266475584B >>>>>>> TotalSeconds: 3285.826 >>>>>>> QPS: 18.943MB/S >>>>>>> >>>>>>> table schema: create table foo (name, age, sex); >>>>>>> >>>>>>> 3. hypertable.cfg >>>>>>> # HDFS Broker >>>>>>> HdfsBroker.Port=38030 >>>>>>> HdfsBroker.fs.default.name=hdfs://dev1:9000 >>>>>>> HdfsBroker.Workers=20 >>>>>>> # Hyperspace >>>>>>> Hyperspace.Replica.Host=localhost >>>>>>> Hyperspace.Replica.Port=38040 >>>>>>> Hyperspace.Replica.Dir=hyperspace >>>>>>> Hyperspace.Replica.Workers=20 >>>>>>> >>>>>>> # Hypertable.Master >>>>>>> Hypertable.Master.Host=localhost >>>>>>> Hypertable.Master.Port=38050 >>>>>>> Hypertable.Master.Workers=20 >>>>>>> >>>>>>> # Hypertable.RangeServer >>>>>>> Hypertable.RangeServer.Port=38060 >>>>>>> Hypertable.RangeServer.MemoryLimit.Percentage=60 >>>>>>> Hypertable.RangeServer.CommitLog.PruneThreshold.Min=50000000 >>>>>>> Hypertable.RangeServer.CommitLog.PruneThreshold.Max=100000000 >>>>>>> Hypertable.RangeServer.Timer.Interval=10000 >>>>>>> Hypertable.RangeServer.Maintenance.Interval=15000 >>>>>>> >>>>>>> Hyperspace.KeepAlive.Interval=30000 >>>>>>> Hyperspace.Lease.Interval=1000000 >>>>>>> Hyperspace.GracePeriod=200000 >>>>>>> >>>>>>> >>>>>>> >>>>>>> 4. memory status: (from Hypertable.RangerServer.log) >>>>>>> >>>>>>> STAT ***** MaintenanceScheduler::schedule() ***** >>>>>>> STAT revision_root 1302839066478990001 >>>>>>> STAT revision_metadata 1302839108088203001 >>>>>>> STAT revision_user 1302844927547261001 >>>>>>> STAT 0/0[..0/0:��](location) cumulative_size 137 <= prune_threshold >>>>>>> 50000000 >>>>>>> STAT 0/0[0/0:��..��](default) cumulative_size 28635 <= >>>>>>> prune_threshold 50000000 >>>>>>> STAT 0/0[0/0:��](location) cumulative_size 28635 <= prune_threshold >>>>>>> 50000000 >>>>>>> STATS user log prune threshold 50000000 >>>>>>> STAT 1/0[..row_key_of_302072](default) cumulative_size 40819432 <= >>>>>>> prune_threshold 50000000 >>>>>>> STAT 1/0[row_key_of_74178..��](default) cumulative_size 40819432 <= >>>>>>> prune_threshold 50000000 >>>>>>> 1302845608 INFO Hypertable.RangeServer : >>>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2817) >>>>>>> Memory Usage: 216582972 bytes >>>>>>> 1302845618 INFO Hypertable.RangeServer : >>>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:84) >>>>>>> Maintenance stats scans=(0 0 0 0.000000) updates=(0 0 0 0.000000 0) >>>>>>> 1302845618 INFO Hypertable.RangeServer : >>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:255) Purging >>>>>>> commit >>>>>>> log fragments with latest revision older than 1302839066478990001 >>>>>>> 1302845618 INFO Hypertable.RangeServer : >>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:255) Purging >>>>>>> commit >>>>>>> log fragments with latest revision older than 1302839108088203001 >>>>>>> 1302845618 INFO Hypertable.RangeServer : >>>>>>> (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:255) Purging >>>>>>> commit >>>>>>> log fragments with latest revision older than 1302844927547261001 >>>>>>> 1302845618 INFO Hypertable.RangeServer : >>>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:171) >>>>>>> Memory Statistics (MB): VM=10259.90, RSS=8223.74, tracked=216.58, >>>>>>> computed=216.58 limit=8424.00 >>>>>>> 1302845618 INFO Hypertable.RangeServer : >>>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:176) >>>>>>> Memory Allocation: BlockCache=0.00% BlockIndex=24.58% BloomFilter=0.05% >>>>>>> CellCache=52.29% ShadowCache=0.00% QueryCache=23.09% >>>>>>> >>>>>>> >>>>>>> 5. I do not use ThriftBroker, I write my own HypertableClient (c++) >>>>>>> according to the implementation of ThriftBroker.cpp >>>>>>> HypertableClient.cpp HypertableClient.h HypertableTypes.h was in >>>>>>> attachment. >>>>>>> >>>>>>> >>>>>>> 6. Here is my test code: PressClient.cpp >>>>>>> >>>>>>> #include "HypertableClient.h" >>>>>>> #include <iostream> >>>>>>> #include <vector> >>>>>>> #include <pca/util/log/log.h> >>>>>>> #include <signal.h> >>>>>>> >>>>>>> using namespace std; >>>>>>> using namespace htclient; >>>>>>> using namespace pca; >>>>>>> >>>>>>> int64_t totalBytes = 0; >>>>>>> int64_t totalUsec = 0; >>>>>>> >>>>>>> void sighandler(int signo) >>>>>>> { >>>>>>> double totalSec = totalUsec*1.0/(1000*1000); >>>>>>> printf("TotalBytes: %ldB\n", totalBytes); >>>>>>> printf("TotalSeconds: %.3lf\n", totalSec); >>>>>>> printf("QPS: %.3fMB/S\n", (totalBytes*1.0/totalSec)/(1024*1024)); >>>>>>> } >>>>>>> >>>>>>> int main(int argc, char *argv[]) >>>>>>> { >>>>>>> if (argc != 4){ >>>>>>> printf("usage: %s type(0:insert, 1:get) start count\n", >>>>>>> argv[0]); >>>>>>> return -1; >>>>>>> } >>>>>>> int type = atoi(argv[1]); >>>>>>> int start = atoi(argv[2]); >>>>>>> int count = atoi(argv[3]); >>>>>>> >>>>>>> printf("type=%d start=%d count=%d\n", type, start, count); >>>>>>> >>>>>>> >>>>>>> PCA_LOG_SETUP("./log4cpp.conf"); >>>>>>> string config_file = "./hypertable.cfg"; >>>>>>> >>>>>>> HypertableClient* client = >>>>>>> HypertableClient::get_instance(config_file); >>>>>>> if (client == NULL){ >>>>>>> cout << "get HypertaleClient error!" << endl; >>>>>>> return -1; >>>>>>> }else >>>>>>> cout << "get HypertableClient ok!" << endl; >>>>>>> >>>>>>> string ns = "combo_test"; >>>>>>> int64_t nsid = client->open_namespace(ns); >>>>>>> if (nsid <= 0){ >>>>>>> cout << "open ns 'combo_test' error. nsid = " << nsid << >>>>>>> endl; >>>>>>> return -1; >>>>>>> } >>>>>>> >>>>>>> signal(SIGTERM, sighandler); >>>>>>> >>>>>>> string table_name = "foo"; >>>>>>> >>>>>>> if (type == 0){ >>>>>>> int64_t mutator_id = client->open_mutator(nsid, table_name); >>>>>>> if (mutator_id < 0){ >>>>>>> cout << "open mutator error, mutator_id = " << mutator_id >>>>>>> << endl; >>>>>>> return -1; >>>>>>> } >>>>>>> cout << "open mutator ok, mutator_id = " << mutator_id << >>>>>>> endl; >>>>>>> >>>>>>> string base_rowkey= "row_key_of"; >>>>>>> string base_name, tmp_base = "name_value_of"; //13B >>>>>>> for (int i = 0; i < 5000; i++) >>>>>>> base_name += tmp_base; >>>>>>> >>>>>>> char *buf = (char *)malloc(102400); //100KB >>>>>>> memset(buf, 0, 102400); >>>>>>> >>>>>>> struct timeval stv, etv; >>>>>>> for (int i = start; i < start + count; i++){ >>>>>>> gettimeofday(&stv, NULL); >>>>>>> vector<Cell> cells; >>>>>>> sprintf(buf, "%s_%d", base_rowkey.c_str(), i); >>>>>>> string row_key = buf; >>>>>>> for (int j = 0; j < 3; j++){ >>>>>>> Cell cell; >>>>>>> cell.row_key = row_key; >>>>>>> //printf("row_key: %s\n", cell.row_key.c_str()); >>>>>>> string value; >>>>>>> if (j == 0){ >>>>>>> cell.column_family = "name"; >>>>>>> sprintf(buf, "%s_%d", base_name.c_str(), i); >>>>>>> value = buf; >>>>>>> }else if (j == 1){ >>>>>>> cell.column_family = "age"; >>>>>>> sprintf(buf, "%d", 1 + i % 100); >>>>>>> value = buf; >>>>>>> }else if (j == 2){ >>>>>>> cell.column_family = "sex"; >>>>>>> value = (i % 2 ? "male" : "female"); >>>>>>> } >>>>>>> >>>>>>> cell.column_qualifier = buf; >>>>>>> cell.set_value((uint8_t*)value.c_str(), >>>>>>> value.length()+1); >>>>>>> cell.flag = htclient::KeyFlag::INSERT; >>>>>>> cells.push_back(cell); >>>>>>> memset(buf, 0, 102400); >>>>>>> >>>>>>> totalBytes += (sizeof(cell) + >>>>>>> cell.column_family.length() + value.length() + row_key.length()); >>>>>>> //printf("totalBytes: %ld\n", totalBytes); >>>>>>> } >>>>>>> >>>>>>> if (client->set_cells(mutator_id, cells)) >>>>>>> printf("insert ok for row: %s\n", row_key.c_str()); >>>>>>> else >>>>>>> printf("****insert row error*****\n"); >>>>>>> >>>>>>> usleep(2000); >>>>>>> gettimeofday(&etv, NULL); >>>>>>> int32_t usec = (etv.tv_sec - stv.tv_sec)*1000000 + >>>>>>> (etv.tv_usec - stv.tv_usec); >>>>>>> >>>>>>> totalUsec += usec; >>>>>>> >>>>>>> >>>>>>> } >>>>>>> >>>>>>> client->close_mutator(mutator_id); >>>>>>> free(buf); >>>>>>> >>>>>>> kill(getpid(), SIGTERM); >>>>>>> >>>>>>> }else { >>>>>>> //string table_name = "foo"; >>>>>>> htclient::ScanSpec ss; >>>>>>> ss.columns.push_back("name"); >>>>>>> >>>>>>> int64_t scanner_id = client->open_scanner(nsid, table_name, >>>>>>> ss); >>>>>>> >>>>>>> if (scanner_id <= 0){ >>>>>>> cout << "create scanner for table " << table_name << " >>>>>>> error, ret = " << scanner_id << endl; >>>>>>> return -1; >>>>>>> } >>>>>>> cout << "create scanner ok for table " << table_name << endl; >>>>>>> >>>>>>> htclient::Cell cell; >>>>>>> while(client->next_cell(scanner_id, cell) == HTE_OK){ >>>>>>> cout << "-------------get a cell------------- " << endl; >>>>>>> //cout << cell << endl; >>>>>>> cell.clear(); >>>>>>> } >>>>>>> >>>>>>> client->close_scanner(scanner_id); >>>>>>> } >>>>>>> >>>>>>> PCA_LOG_SHUTDOWN(""); >>>>>>> return 0; >>>>>>> } >>>>>>> >>>>>>> >>>>>>> 8. Promblem: the memory always increase when I insert data; when the >>>>>>> insert finished, the RSS alawy 8GB, but RangerServer declared it only >>>>>>> used >>>>>>> 200M. >>>>>>> what the remain memory going? >>>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Hypertable Development" group. >>>>>>> To post to this group, send email to [email protected] >>>>>>> . >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected]. >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/hypertable-dev?hl=en. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
foo.cc
Description: Binary data
