Which hadoop version are you on? 

> -----Original Message-----
> From: bhupesh bansal [mailto:[EMAIL PROTECTED] 
> Sent: Friday, April 11, 2008 11:21 PM
> To: [EMAIL PROTECTED]
> Subject: Mapper OutOfMemoryError Revisited !!
> 
> 
> Hi Guys, I need to restart discussion around 
> http://www.nabble.com/Mapper-Out-of-Memory-td14200563.html
> 
>  I saw the same OOM error in my map-reduce job in the map phase. 
> 
> 1. I tried changing mapred.child.java.opts (bumped to 600M) 
> 2. io.sort.mb was kept at 100MB. 
> 
> I see the same errors still. 
> 
> I checked with debug the size of "keyValBuffer" in collect(), 
> that is always less than io.sort.mb and is spilled to disk properly.
> 
> I tried changing the map.task number to a very high number so 
> that the input is split into smaller chunks. It helps for a 
> while as the map job went a bit far (56% from 5%) but still 
> see the problem.
> 
>  I tried bumping mapred.child.java.opts to 1000M , still got 
> the same error. 
> 
> I also tried using the -verbose:gc -Xloggc:/tmp/@[EMAIL PROTECTED] 
> value in opts to get the gc.log but didnt got any log??
> 
>  I tried using 'jmap -histo pid' to see the heap information, 
> it didnt gave me any meaningful or obvious problem point. 
> 
> What are the other possible memory hog during mapper phase ?? 
> Is the input file chunk kept fully in memory ?? 
> 
> Application: 
> 
> My map-reduce job is running with about 2G of input. in the 
> Mapper phase I read each line and output [5-500] (key,value) 
> pair. so the intermediate data should be really blown up.  
> will that be a problem. 
> 
> The Error file is attached
> http://www.nabble.com/file/p16628181/error.txt error.txt
> --
> View this message in context: 
> http://www.nabble.com/Mapper-OutOfMemoryError-Revisited-%21%21
> -tp16628181p16628181.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
> 
> 

Reply via email to