[
https://issues.apache.org/jira/browse/PIG-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578487#comment-13578487
]
Rohini Palaniswamy commented on PIG-3148:
-----------------------------------------
The problem with the ratio of max heap will hit the same issue as Koji's first
point. In the mapper, the heap is going to grow huge and GC will not do
anything. So triggering GC on ratio of max heap will not work. One alternative
is to check for memory freed after the initial GC and if it does not free more
than some X MB, then turn off the initial GC globally in the
SpillableMemoryManager. Koji was not enthusiastic about making JVM calls to
determine free memory.
> OutOfMemory exception while spilling stale DefaultDataBag. Extra option to
> gc() before spilling large bag.
> ----------------------------------------------------------------------------------------------------------
>
> Key: PIG-3148
> URL: https://issues.apache.org/jira/browse/PIG-3148
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Attachments: pig-3148-v01.patch
>
>
> Our user reported that one of their jobs in pig 0.10 occasionally failed with
> 'Error: GC overhead limit exceeded' or 'Error: Java heap space', but
> rerunning it sometimes finishes successfully.
> For 1G heap reducer, heap dump showed it contained two huge DefaultDataBag
> with 300-400MBytes each when failing with OOM.
> Jstack at the time of OOM always showed that spill was running.
> {noformat}
> "Low Memory Detector" daemon prio=10 tid=0xb9c11800 nid=0xa52 runnable
> [0xb9afc000]
> java.lang.Thread.State: RUNNABLE
> at java.io.FileOutputStream.writeBytes(Native Method)
> at java.io.FileOutputStream.write(FileOutputStream.java:260)
> at
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
> - locked <0xe57c6390> (a java.io.BufferedOutputStream)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> - locked <0xe57c60b8> (a java.io.DataOutputStream)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
> at org.apache.pig.data.utils.SedesHelper.writeBytes(SedesHelper.java:46)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:537)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
> at
> org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
> at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
> at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
> at org.apache.pig.data.DefaultDataBag.spill(DefaultDataBag.java:106)
> - locked <0xceb16190> (a java.util.ArrayList)
> at
> org.apache.pig.impl.util.SpillableMemoryManager.handleNotification(SpillableMemoryManager.java:243)
> - locked <0xbeb86318> (a java.util.LinkedList)
> at
> sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:138)
> at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171)
> at
> sun.management.MemoryPoolImpl$PoolSensor.triggerAction(MemoryPoolImpl.java:272)
> at sun.management.Sensor.trigger(Sensor.java:120)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
