Kind of curious what it's using all that RAM for. Don't suppose you profiled it?
On Mon, Jun 18, 2012 at 1:37 PM, Pankaj Gupta <[email protected]> wrote: > Awesome, that worked. Thanks. > To summarize the root cause and resolution as I understand it: > At the end of first job, PIG did a list on the job and apparently that shot > its Heap size requirement up to 1.7G. The existing, default, heapsize for the > PIG JVM didn't allow that and gave an out of memory error. Setting > HADOOP_HEAPSIZE to 8192 set the max heap size of the PIG process to 8196m and > the problem went away. > > Thanks, > Pankaj > On Jun 18, 2012, at 1:23 AM, Aniket Mokashi wrote: > >> export HADOOP_HEAPSIZE=<something more than what it is right now> >> >> Thanks, >> Aniket >> >> On Sun, Jun 17, 2012 at 11:16 PM, Pankaj Gupta <[email protected]>wrote: >> >>> Hi, >>> >>> I am getting an out of memory error while running Pig. I am running a >>> pretty big job with one master node and over 100 worker nodes. Pig divides >>> the execution in two map-reduce jobs. Both the jobs succeed with a small >>> data set. With a large data set I get an out of memory error at the end of >>> the first job. This happens right after the all the mappers and reducers of >>> the first job are done and the second job hasn't started. Here is the error: >>> >>> 2012-06-18 03:15:29,565 [Low Memory Detector] INFO >>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler >>> call - Collection threshold init = 187039744(182656K) used = >>> 390873656(381712K) committed = 613744640(599360K) max = 699072512(682688K) >>> 2012-06-18 03:15:31,137 [Low Memory Detector] INFO >>> org.apache.pig.impl.util.SpillableMemoryManager - first memory handler >>> call- Usage threshold init = 187039744(182656K) used = 510001720(498048K) >>> committed = 613744640(599360K) max = 699072512(682688K) >>> Exception in thread "IPC Client (47) connection to /10.217.23.253:9001from >>> hadoop" java.lang.RuntimeException: >>> java.lang.reflect.InvocationTargetException >>> Caused by: java.lang.reflect.InvocationTargetException >>> Caused by: java.lang.OutOfMemoryError: Java heap space >>> at org.apache.hadoop.mapred.TaskReport.<init>(TaskReport.java:46) >>> at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown >>> Source) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113) >>> at >>> org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:53) >>> at >>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:236) >>> at >>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:171) >>> at >>> org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:219) >>> at >>> org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:66) >>> at >>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:807) >>> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:742) >>> Exception in thread "Low Memory Detector" java.lang.OutOfMemoryError: Java >>> heap space >>> at >>> sun.management.MemoryUsageCompositeData.getCompositeData(MemoryUsageCompositeData.java:40) >>> at >>> sun.management.MemoryUsageCompositeData.toCompositeData(MemoryUsageCompositeData.java:34) >>> at >>> sun.management.MemoryNotifInfoCompositeData.getCompositeData(MemoryNotifInfoCompositeData.java:42) >>> at >>> sun.management.MemoryNotifInfoCompositeData.toCompositeData(MemoryNotifInfoCompositeData.java:36) >>> at sun.management.MemoryImpl.createNotification(MemoryImpl.java:168) >>> at >>> sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:300) >>> at sun.management.Sensor.trigger(Sensor.java:120) >>> >>> I will really appreciate and suggestions on how to go about debugging and >>> rectifying this issue. >>> >>> Thanks, >>> Pankaj >> >> >> >> >> -- >> "...:::Aniket:::... Quetzalco@tl" >
