MR processes' Child JVMs run as the 'mapred' user in a non-secure cluster (i.e. they run as the same user as the one that runs the TaskTracker parent). Ensure the mapred user too has its default virtual memory raised.
Also, for Child JVMs, the job config "mapred.child.ulimit" is applied, so you will want to check that value's settings too. In case you have a valid bug despite these two additional points you need to consider, please report it onto https://issues.cloudera.org/browse/DISTRO (Btw, moved discussion to CDH users list cdh-u...@cloudera.org, since it may be CDH specific as you say) On Thu, Jun 14, 2012 at 4:11 AM, Yang <teddyyyy...@gmail.com> wrote: > the Child of our jobs are launched using user hdfs > > we got OOM in some jobs when they try to mmap() a big file. > > I manually checked the ulimit -v of hdfs on the slave nodes, they are > unlimited, > > but when I launched a fake streaming job, whose mapper is simply bash -c ' > ulimit -a ', > I found that it has a 1G virtual memory limit. > > > why is this?? > > > looks like a Cloudera bug > > we are using CDH3U3 > > > Thanks > Yang -- Harsh J
