Hi Chris, Thanks for getting back to me. I will set that value to 10
I have just tried this. https://support.gopivotal.com/hc/en-us/articles/201462036-Mapreduce-YARN-Me mory-Parameters Setting both to mapreduce.map.memory.mb mapreduce.reduce.memory.mb. Though after setting it I didn’t get the expected change. As the output was still 2.1 GB of 2.1 GB virtual memory used. Killing container Regards, Chris MacKenzie telephone: 0131 332 6967 email: [email protected] corporate: www.chrismackenziephotography.co.uk <http://www.chrismackenziephotography.co.uk/> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts> <http://www.linkedin.com/in/chrismackenziephotography/> From: Chris Mawata <[email protected]> Reply-To: <[email protected]> Date: Thursday, 17 July 2014 13:36 To: Chris MacKenzie <[email protected]> Cc: <[email protected]> Subject: Re: Configuration set up questions - Container killed on request. Exit code is 143 Hi Chris MacKenzie, I have a feeling (I am not familiar with the kind of work you are doing) that your application is memory intensive. 8 cores per node and only 12GB is tight. Try bumping up the yarn.nodemanager.vmem-pmem-ratio Chris Mawata On Wed, Jul 16, 2014 at 11:37 PM, Chris MacKenzie <[email protected]> wrote: Hi, Thanks Chris Mawata I’m working through this myself, but wondered if anyone could point me in the right direction. I have attached my configs. I’m using hadoop 2.41 My system is: 32 Clusters 8 processors per machine 12 gb ram Available disk space per node 890 gb This is my current error: mapreduce.Job (Job.java:printTaskEvents(1441)) - Task Id : attempt_1405538067846_0006_r_000000_1, Status : FAILED Container [pid=25848,containerID=container_1405538067846_0006_01_000004] is running beyond virtual memory limits. Current usage: 439.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1405538067846_0006_01_000004 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 25853 25848 25848 25848 (java) 2262 193 2268090368 112050 /usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx768m -Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog s/application_1405538067846_0006/container_1405538067846_0006_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056 attempt_1405538067846_0006_r_000000_1 4 |- 25848 25423 25848 25848 (bash) 0 0 108613632 333 /bin/bash -c /usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx768m -Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog s/application_1405538067846_0006/container_1405538067846_0006_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056 attempt_1405538067846_0006_r_000000_1 4 1>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846 _0006/container_1405538067846_0006_01_000004/stdout 2>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846 _0006/container_1405538067846_0006_01_000004/stderr Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 Regards, Chris MacKenzie telephone: 0131 332 6967 email: [email protected] corporate: www.chrismackenziephotography.co.uk <http://www.chrismackenziephotography.co.uk> <http://www.chrismackenziephotography.co.uk/> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts> <http://www.linkedin.com/in/chrismackenziephotography/> From: Chris Mawata <[email protected]> Reply-To: <[email protected]> Date: Thursday, 17 July 2014 02:10 To: <[email protected]> Subject: Re: Can someone shed some light on this ? - java.io.IOException: Spill failed I would post the configuration files -- easier for someone to spot something wrong than to imagine what configuration would get you to that stacktrace. The part Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1405523201400_0006_m_000000_0_spill_8.out would suggest you might not have hadoop.tmp.dir set (?) On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie <[email protected]> wrote: Hi, Is this a coding or a setup issue ? I¹m using Hadoop 2.41 My program is doing a concordance on 500,000 sequences of 400 chars. My cluster set is 32 data nodes and two masters. The exact error is: Error: java.io.IOException: Spill failed at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas k.java:1535) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput OutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp er.java:112) at par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: 96) at par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java: 1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j ava:1556) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for attempt_1405523201400_0006_m_000000_0_spill_8.out at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW rite(LocalDirAllocator.java:402) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat or.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat or.java:131) at org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil es.java:159) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java: 1566) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85 3) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja va:1505) Regards, Chris
