Thanks for the reply Srinivas, so option 2 will be enough, however, when I tried setting it to 512MB, I see through the system monitor that the map task is given 275MB of real memory!! Is that normal in hadoop to go over the upper bound of memory given by the property mapred.child.java.opts.
Mark On Wed, Feb 15, 2012 at 4:00 PM, Srinivas Surasani <[email protected]> wrote: > Hey Mark, > > Yes, you can limit the memory for each task with > "mapred.child.java.opts" property. Set this to final if no developer > has to change it . > > Little intro to "mapred.task.default.maxvmem" > > This property has to be set on both the JobTracker for making > scheduling decisions and on the TaskTracker nodes for the sake of > memory management. If a job doesn't specify its virtual memory > requirement by setting mapred.task.maxvmem to -1, tasks are assured a > memory limit set to this property. This property is set to -1 by > default. This value should in general be less than the cluster-wide > configuration mapred.task.limit.maxvmem. If not or if it is not set, > TaskTracker's memory management will be disabled and a scheduler's > memory based scheduling decisions may be affected. > > On Wed, Feb 15, 2012 at 5:57 PM, Mark question <[email protected]> > wrote: > > Hi, > > > > My question is what's the difference between the following two settings: > > > > 1. mapred.task.default.maxvmem > > 2. mapred.child.java.opts > > > > The first one is used by the TT to monitor the memory usage of tasks, > while > > the second one is the maximum heap space assigned for each task. I want > to > > limit each task to use upto say 100MB of memory. Can I use only #2 ?? > > > > Thank you, > > Mark > > > > -- > -- Srinivas > [email protected] >
