On 7/1/08 3:20 PM, "Mori Bellamy" <[EMAIL PROTECTED]> wrote:
> i've got a mapreduce task that works on small (~1G) input. when i try
> to run the same task on large (~100G) input, i get the following error
> around when the map tasks are almost done (~98%)
[error list deleted]
> I'm running the task on a cluster of 5 workers, one DFS master, and
> one task tracker. i'm chaining mapreduce tasks, so i'm using
> SequenceFileOutput and SequenceFileInput. this error happens before
> the first link in the chain sucessfully reduces.
>
> does anyone have any insight? thanks!
Any chance your tasks are running out of memory? I've seen similar
errors when we had our memory watchdog set too low and the tasks were killed
during the shuffle. Woops. :)
