Sorry, I sent out an incomplete message. The last sentence should be: You can request a larger heap for Map tasks by including -Xmx in mapred.child.java.opts
regards, Sriguru >-----Original Message----- >From: Srigurunath Chakravarthi >Sent: Monday, February 08, 2010 11:51 PM >To: '[email protected]' >Subject: RE: heap memory > >Hi Gang, > Not sure if I understood your question right. Responses inline: > >>-----Original Message----- >>From: Gang Luo [mailto:[email protected]] >>Sent: Friday, February 05, 2010 8:24 AM >>To: [email protected] >>Subject: heap memory >> >>HI all, >>I suppose there is only map function that will consume the heap memory >>assigned to each map task. While the default heap memory is 200 mb, I just >>wonder most of the memory is wasted for a simple map function (e.g. >>IdentityMapper). > >Map Output records get buffered before being spilt to disk. You can control >this size via io.sort.mb. > >> >>So, I try to make use of this memory by buffering the output records, or >>maintaining large data structure in memory, but it doesn't work as I >>expect. For example, I want to build a hash table on a 100mb table in >>memory during the life time of that map task. But it fails due to lack of >>heap memory. Don't I get 200mb heap memory? What others also eat my heap >>memory? > >You can request a larger head for Map tasks by including -Xmx > >> >>Thanks. >>-Gang >> >> >> >> ___________________________________________________________ >> 好玩贺卡等你发，邮箱贺卡全新上线！ >>http://card.mail.cn.yahoo.com/
