[ https://issues.apache.org/jira/browse/PIG-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheolsoo Park updated PIG-3101: ------------------------------- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. > Increase io.sort.mb in YARN MiniCluster > --------------------------------------- > > Key: PIG-3101 > URL: https://issues.apache.org/jira/browse/PIG-3101 > Project: Pig > Issue Type: Bug > Affects Versions: 0.11 > Reporter: Cheolsoo Park > Assignee: Cheolsoo Park > Fix For: 0.12 > > Attachments: PIG-3101.patch > > > I see some unit test cases intermittently fail with the following error with > Hadoop-2.0.x: > {code} > 2012-12-17 18:28:39,235 FATAL [Low Memory Detector] > org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Low Memory > Detector,9,system] threw an Error. Shutting down now... > java.lang.InternalError: Error in invoking listener > at > sun.management.NotificationEmitterSupport.sendNotification(NotificationEmitterSupport.java:141) > at sun.management.MemoryImpl.createNotification(MemoryImpl.java:171) > at > sun.management.MemoryPoolImpl$CollectionSensor.triggerAction(MemoryPoolImpl.java:300) > at sun.management.Sensor.trigger(Sensor.java:120) > 2012-12-17 18:28:39,285 INFO [main] org.apache.hadoop.mapred.MapTask: > Finished spill 0 > 2012-12-17 18:28:39,342 WARN [main] org.apache.hadoop.mapred.YarnChild: > Exception running child : java.lang.IllegalStateException: Shutdown in > progress, cannot add a shutdownHook > at > org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152) > > at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2296) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2268) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:302) > at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:273) > at org.apache.hadoop.mapred.SpillRecord.writeToFile(SpillRecord.java:124) > at org.apache.hadoop.mapred.SpillRecord.writeToFile(SpillRecord.java:119) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1760) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1440) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:664) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:732) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:154) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:149) > {code} > By default, {{mapred.io.sort.mb}} is set to {{100}}, and > {{mapred.child.java.opts}} is set to {{-Xmx200m}}. Increasing them to {{200}} > and {{512m}} respectively makes the intermittent test failures go away. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira