It turns out the hadoop-default.xml packaged in my custom hadoop-0.19.0-core.jar has set the "io.file.buffer.size" to 131072 (128KB), which means DfsBroker has to open a 128KB buffer for every open file. The official hadoop-0.19.0-core.jar sets this value to 4096, which is more reasonable for applications like Hypertable. Donald
On Fri, Feb 20, 2009 at 11:55 AM, Liu Kejia (Donald) <[email protected]>wrote: > Caching might not work very well because keys are randomly generated, > resulting in bad locality... > Even it's Java, hundreds of kilobytes per file object is still very big. > I'll profile HdfsBroker to see what exactly is using so much memory, and > post the results later. > > Donald > > > On Fri, Feb 20, 2009 at 11:20 AM, Doug Judd <[email protected]> wrote: > >> Hi Donald, >> >> Interesting. One possibility would be to have an open CellStore cache. >> Frequently accessed CellStores would remain open, while seldom used ones get >> closed. The effectiveness of this solution would depend on the workload. >> Do you think this might work for your use case? >> >> - Doug >> >> >> On Thu, Feb 19, 2009 at 7:09 PM, donald <[email protected]> wrote: >> >>> >>> Hi all, >>> >>> I recently run into the problem that HdfsBroker throws out of memory >>> exception, because too many CellStore files in HDFS are kept open - I >>> have over 600 ranges per range server, with a maximum of 10 cell >>> stores per range, that'll be 6,000 open files at the same time, making >>> HdfsBroker to take gigabytes of memory. >>> >>> If we open the CellStore file on demand, i.e. when a scanner is >>> created on it, this problem is gone. However random-read performance >>> may drop due to the the overhead of opening a file in HDFS. Any better >>> solution? >>> >>> Donald >>> >>> >> >> >> >> > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
