Awesome! I think that may have fixed it. Will have to see the final results tomorrow...
On Sat, Sep 18, 2010 at 3:47 PM, Chris K Wensel <[email protected]> wrote: > see if this helps matters > > --bootstrap-action > s3://elasticmapreduce/bootstrap-actions/create-swap-file.rb --args > "-E,/mnt/swap,1000" > > ckw > > On Sep 18, 2010, at 1:22 PM, Bradford Stephens wrote: > > > That could be a problem... this is Elastic MapReduce. Can I do that? > Guess > > I'll have to experiment. > > > > Cheers, > > B > > > > On Sat, Sep 18, 2010 at 7:21 AM, Edward Capriolo <[email protected] > >wrote: > > > >> This happens because child processes try to allocate the same memory > >> as the parent. One way to solve this is setting memory overcommit to > >> your linux system. > >> > >> On Sat, Sep 18, 2010 at 4:47 AM, Bradford Stephens > >> <[email protected]> wrote: > >>> Hey guys, > >>> > >>> I'm running into issues when doing a moderate-size EMR job on 12 > m1.large > >>> nodes. Mappers and Reducers will randomly fail. > >>> > >>> The EMR defaults are 2 mappers / 2 reducers per node. I've tried > running > >>> with mapred.child.opts set in the jobconf to -Xmx256m and -Xmx1024m. No > >>> difference. > >>> > >>> There are about 1,000 map tasks. Not very much data, maybe 50G at most? > >>> > >>> My job fails to complete. Looking in syslog shows this: > >>> > >>> java.io.IOException: Cannot run program "bash": java.io.IOException: > >>> error=12, Cannot allocate memory > >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) > >>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) > >>> at org.apache.hadoop.util.Shell.run(Shell.java:134) > >>> at org.apache.hadoop.fs.DF.getAvailable(DF.java:73) > >>> at > >>> > >> > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:329) > >>> at > >>> > >> > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) > >>> at > >>> > >> > org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107) > >>> at > >>> > >> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1238) > >>> at > >>> > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1146) > >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:365) > >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:312) > >>> at org.apache.hadoop.mapred.Child.main(Child.java:170) > >>> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot > >>> allocate memory > >>> at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) > >>> at java.lang.ProcessImpl.start(ProcessImpl.java:65) > >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) > >>> ... 11 more > >>> > >>> > >>> > >>> I would ask the EMR forums, but I think I may get faster feedback here > :) > >>> > >>> > >>> -- > >>> Bradford Stephens, > >>> Founder, Drawn to Scale > >>> drawntoscalehq.com > >>> 727.697.7528 > >>> > >>> http://www.drawntoscalehq.com -- The intuitive, cloud-scale data > >> solution. > >>> Process, store, query, search, and serve all your data. > >>> > >>> http://www.roadtofailure.com -- The Fringes of Scalability, Social > >> Media, > >>> and Computer Science > >>> > >> > > > > > > > > -- > > Bradford Stephens, > > Founder, Drawn to Scale > > drawntoscalehq.com > > 727.697.7528 > > > > http://www.drawntoscalehq.com -- The intuitive, cloud-scale data > solution. > > Process, store, query, search, and serve all your data. > > > > http://www.roadtofailure.com -- The Fringes of Scalability, Social > Media, > > and Computer Science > > -- > Chris K Wensel > [email protected] > http://www.concurrentinc.com > > -- Concurrent, Inc. offers mentoring, support, and licensing for Cascading > > -- Bradford Stephens, Founder, Drawn to Scale drawntoscalehq.com 727.697.7528 http://www.drawntoscalehq.com -- The intuitive, cloud-scale data solution. Process, store, query, search, and serve all your data. http://www.roadtofailure.com -- The Fringes of Scalability, Social Media, and Computer Science
