On Fri, Jun 27, 2008 at 08:57, Chris Anderson <[EMAIL PROTECTED]> wrote:
> The problem is that when there are a large number of map tasks to
> complete, Hadoop doesn't seem to obey the map.tasks.maximum. Instead,
> it is spawning 8 map tasks per tasktracker (even when I change the
> mapred.tasktra
Having experimented some more, I've found that the simple solution is
to limit the resource usage by limiting the # of map tasks and the
memory they are allowed to consume.
I'm specifying the constraints on the command line like this:
-jobconf mapred.tasktracker.map.tasks.maximum=2 mapred.child.u
Allen Wittenauer wrote:
This is essentially what we're doing via torque (and therefore hod).
When we intend to move away from HOD(and thus torque) to using Hadoop
Resource manager(HADOOP-3421) and scheduler(HADOOP-3412) interfaces, we
need to move this resource management functionality into
On 6/26/08 10:14 AM, "Joydeep Sen Sarma" <[EMAIL PROTECTED]> wrote:
> However - in our environment - we spawn all streaming tasks through a
> wrapper program (with the wrapper being defaulted across all users). We
> can control resource uses from this wrapper (and also do some level of
> job contro
Memory limits were handled in this jira:
https://issues.apache.org/jira/browse/HADOOP-2765
However - in our environment - we spawn all streaming tasks through a
wrapper program (with the wrapper being defaulted across all users). We
can control resource uses from this wrapper (and also do some lev
