Hi Gang, Not sure if I understood your question right. Responses inline: >-----Original Message----- >From: Gang Luo [mailto:[email protected]] >Sent: Friday, February 05, 2010 8:24 AM >To: [email protected] >Subject: heap memory > >HI all, >I suppose there is only map function that will consume the heap memory >assigned to each map task. While the default heap memory is 200 mb, I just >wonder most of the memory is wasted for a simple map function (e.g. >IdentityMapper).
Map Output records get buffered before being spilt to disk. You can control this size via io.sort.mb. > >So, I try to make use of this memory by buffering the output records, or >maintaining large data structure in memory, but it doesn't work as I >expect. For example, I want to build a hash table on a 100mb table in >memory during the life time of that map task. But it fails due to lack of >heap memory. Don't I get 200mb heap memory? What others also eat my heap >memory? You can request a larger head for Map tasks by including -Xmx > >Thanks. >-Gang > > > > ___________________________________________________________ > 好玩贺卡等你发，邮箱贺卡全新上线！ >http://card.mail.cn.yahoo.com/
